ciop-copy (7)¶

Synopsis¶

ciop-copy {url} [option…]

ciop-copy copies to a local folder the enclosure URL (the online resource taken from a catalogue) passed as argument. The inputs may come from:

url: one or more product urls
-h: displays the help page
-a: abort on first error without attempting to process further URLs
-q: quiet mode, local filenames are not echoed to stdout after transfer
-f: force transfer of a physical copy of the file for nfs and HDFS urls.
-d driver-file: get additional drivers from shell file <driver-file>. Drivers shall contain a named <protocol> driver
-o|O <out-dir>: same as -c -o <out-dir>. Kept for retro-compatibility.
-c: creates the output directory if it does not exist
-p prefix: prepend the given prefix to all output names
-z: provide output as a compressed package (.gz for files or .tgz for folders). NOTE that it will not compress already compressed files (.gz, .tgz or .zip)
-U|--no-unzip<prefix>: disable file automatic decompression of .gz, .tgz and .zip files
-rt seconds<prefix>: define the time (in seconds) between retries (default is 60)
-t timeout<prefix>: defines the timer (in seconds) for the watchdog timeout applicable to gridftp, scp, ftp, http, and https schemes (default is 600 seconds)
-R timeout: do not retry transfer after timeout
-D: set debug mode for command output parsing debugging
-s: skip download if sink path already exists
-x<pattern>: exclude the files matching the pattern
-w<pattern>: do not overwrite single files iif already exist

Unless the -q option is used (quiet mode), the local path of each file or directory downloaded after each URL transfer is echoed, one per line.

Unless the -U option is used, .gz or .tgz output files will be decompressed.

Unless the -H option is used, the software will follow the RDF <dclite4g:onlineResource> and the HTML href & refresh tags.

ciop-copy command exits

0 all URLs were successfully downloaded
1 an error occured during processing
255 environment is invalid (e.g. invalid working directory) or invalid options are provided
254 output directory does not exist or failed creating it (with -c option)

If the -a option is used, the exit code is set to the error code of the last URL transfer:

252 no driver available for URL
251 an existing file or directory conflicts with the sink for the URL in the output directory
250 an error occured while unpacking the output file or when packaging/compressing the output file (when -z or -Z option is used)
128 a timeout occured while fetching an url
127 a fatal error occured, source of error is not known or not handled by driver
128 error codes specific to the transfer scheme
1 resource pointed by input URL does not exist

Input from opensearch-client:

opensearch-client "${MASTER}" enclosure | ciop-copy -f -O ${UUIDTMP}/data/master -

Input from the use of the source type “cas:series” in the application.xml, e.g.:

Application.xml:

<source id="cas_source" refid="cas:series">http://catalogue.terradue.int/catalogue/search/MER_RR__1P/description</source>

run.sh:

while read product
do
        prod=$( echo $product | ciop-copy -U -o ${TMPDIR}/input - )
done

Terradue