Changes

This [[Module Library|XQuery Module]] provides functions for evaluating XQuery expressions in separate threadsregistering new query jobs and orchestrating existing jobs. Query execution Jobs can both be parallelized queries, commands, operations performed by a database client, and postponed to be executed asynchronouslyHTTP requests.

All functions in this module are assigned to the <code><nowiki>http://basex.org/modules/asyncjob</nowiki></code> namespace, which is statically bound to the {{Code|asyncjob}} prefix. Errors will be bound to the same prefix.

=Parallelized ExecutionServices=

Parallel query execution is recommendable if you have various calls that require a lot of time, but cannot A job can be sped up registered as ''service'' by rewriting supplying the code. This is e. g. the case if external URLs are called. If you are parallelizing local data reads (such as accessing a database), your single-threaded query will usually be faster, because parallelized access {{Code|service}} option to disk data will often lead to randomized access patterns, which can hardly be optimized by your HD or SSD.{{Function||job:eval}}:

<pre lang='xquery'>(: register job as service; will be run every day at 1 am :)job:eval('db:drop("tmp")', (), map { 'id':'cleanup', 'start':'01:00:00', 'interval':'P1D', 'service': true() }), (: list registered services :)job:services(),(: result: <job base-uri="..." id="cleanup" interval="P1D" start=async"01:00:00">db:drop("tmp")</job> :) (: unregister job :)job:remove('cleanup', map { 'service': true() })</pre> '''Some more notes:fork''' * All job services will be scheduled for evaluation when the BaseX server or BaseX HTTP server is started.* If a job service is outdated (e.g. because a supplied end time has been exceeded), it will be removed from the jobs file at startup time.* The job definitions are stored in a {{Code|jobs.xml}} file in the database directory. It can also be edited manually. =Executing Jobs= There are cases in which a client does not, or cannot, wait until a request is fully processed. The client may be a browser, which sends an HTTP request to the server to start another time-joinconsuming query job. The functions in this section allow you to register new query jobs and access existing ones. Jobs can be executed immediately (i.e., as soon as a [[Transaction Management#Concurrency Control|free slot is available]]) or scheduled for repeated execution. Each registered job gets a job ID, and the ID can be used to retrieve a query result, stop a job, or wait for its termination. ==job:eval==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async<pre>job:fork-join|eval( $functions query as function(*)*|item()*}}<br/ >{{Func|asyncxs:fork-join|anyAtomicType, $functions bindings as functionmap(*)*? := map { }, $options as map(xs*)? :string, = map { }) as xs:string)|item()*}}<br/ pre>|-valign="top"|'''Summary'''|This function executes Schedules the evaluation a new query job for the supplied {{Code|$query}} (non-of type {{Code|xs:string}}, or of type {{Code|xs:anyURI}} if points to a resource), and returns a job ID. The job will be queued until a free slot is available, and the query result can be cached. Queries can be updating, and variables and the context value can be declared via {{Code|$bindings}} (see {{Function|XQuery|xquery:eval}} for more details) functions in parallel. The following {{Code|$options}} can be supplied:* {{Code|cache}}: indicates if the query result will be cached or ignored (default: {{Code|false}}):** The result will be cached in main-memory until it is fetched via {{Function||job:result}}, or until {{Option|CACHETIMEOUT}} is exceeded.** If the query raises an error, it will be cached and returned instead.* {{Code|start}}: a dayTimeDuration, time, dateTime or integer can be specified to delay the execution of the query:** If a dayTimeDuration is specified, the query will be queued after the specified duration has passed. Examples of valid values are available:<code>P1D</code> (1 day), <code>PT5M</code> (5 minutes), <code>PT0.1S</code> (100 ms). An error will be raised if a negative value is specified.* * If a dateTime is specified, the query will be executed at this date. Examples for valid values are: <code>threads2018-12-31T23:59:59</code>(New Year's Eve 2018, close to midnight). An error will be raised if the specified time lies in the past.** If a time is specified, the query will be executed at this time of the day. Examples of valid times are: maximum number of parallel threads <code>02:00:00</code> (default2am local time), <code>12:00: available 00Z</code> (noon, UTC). If the time lies in the past, the query will be executed the next day.** An integer will be interpreted as minutes. If the specified number is greater than the elapsed minutes of coresthe current hour, the query will be executed one hour later.* {{Code|interval}}: a dayTimeDuration string can be specified to execute the query periodically. An error is raised if the specified interval is less than one second (<code>PT1S</code>). If the next scheduled call is due, and if a query with the same ID is still running, it will be skipped.* {{Code|end}}: scheduling can be stopped after a given time or duration. The string format is the same as for {{Code|start}}. An error is raised if the resulting end time is smaller than the start time.* {{Code|base-uri}}: sets the [https://www.w3.org/TR/xquery-31/#dt-static-base-uri base-uri property] for the query. This URI will be used when resolving relative URIs, such as with {{Code|fn:doc}}.* {{Code|id}}: sets a custom job ID. The ID must not start with the standard <code>thread-sizejob</code>prefix, and it can only be assigned if no job with the same name exists.* {{Code|service}}: number additionally registers the job as [[#Services|service]]. Registered services must have no variable bindings.* {{Code|log}}: writes the specified string to the [[Logging|database logs]]. Two log entries are stored, one at the beginning and another one after the execution of functions to the job.|- valign="top"| '''Errors'''|{{Error|overflow|#Errors}} Query execution is rejected because too many jobs are queued or being executed. {{Option|CACHETIMEOUT}} can be evaluated by each thread (decreased if the default: setting is too restrictive.<br/>{{Error|range|#Errors}} A specified time or duration is out of range.<codebr/>1{{Error|id|#Errors}} The specified ID is invalid or has already been assigned.<br/code>){{Error|options|#Errors}} The specified options are conflicting.|-valign="top"

* Cache query result. The returned ID can be used to pick up the result with {{Function||job:result}}:<pre lang='xquery'>job:eval("1+3", (), map { 'cache': true() })</pre>* A happy birthday mail will be sent at the given date:<pre lang='xquery'>job:eval("import module namespace mail='mail'; mail:send('Happy birthday!')", (), map { 'start': '2018-09-01T06:00:00' })}}</pre>* The following [[RESTXQ]] functions can be called to execute a query at 2 am every day. An ID will be returned by the first function sleeps in parallel; it will , which can be finished in 1 used to stop the scheduler via the second if your system has at least 2 coresfunction:<pre classlang='brush:xquery'>asyncdeclare %rest:POST("{$query}") %rest:forkpath('/start-joinscheduling') function local:start($query) { for job:eval($i in 1 to 2query, (), map { 'start': '02:00:00', 'interval': 'P1D' })}; return declare %rest:path('/stop-scheduling/{$id}') functionlocal:stop($id) { job:remove($id)};</pre>* Query execution is scheduled for every second, and for 10 seconds in total. As the query itself will take 1.5 seconds, it will only be executed every second time:<pre lang='xquery'>job:eval("prof:sleep(10001500)", (), map { 'interval': 'PT1S', 'end': 'PT10S' }) </pre>* The query in the specified file will be evaluated once:<pre lang='xquery'>job:eval(xs:anyURI('cleanup.xq'))</pre>* The following expression, if stored in a file, will be evaluated every 5 seconds:<pre lang='xquery'>job:eval( static-base-uri(), map { }, map { 'start': 'PT5S' }

=Asynchronous Execution= Asynchronous query execution is recommendable if a client does not, or cannot, wait until a request is fully processed. This is e. g. the case with web browsers, which will usually cancel a request after a specific timeout. In such cases, you can use asynchronous execution to trigger another server-side process, which will start the time-consuming process, and fetch the job:result later on as soon as it is available. ==async:eval==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async:eval|$query as xs:string|xs:string}}<br /pre>{{Func|asyncjob:eval|result( $query id as xs:string, $bindings options as map(*)|xs? :string= map { }}<br />{{Func|async:eval|$query as xs:string, $bindings as map(*), $options as mapitem(xs:string, xs:string)|xs:string}}*<br /pre>|-valign="top"

|Prepares Returns the supplied cached result of a job with the specified job {{Code|$queryid}} string for asynchronous execution and returns a query id:* If the original job has raised an error, the cached error will be raised instead. * The query cached result or error will be queued as described in the article on [[Transaction Management]], and dropped after it has been retrieved.* If the result will be has not been cached in main-memory until or if it is fetched via [[#async:result|async:result]]has been dropped, or until {{Option|ASYNCTIMEOUT}} an empty sequence is exceeded.<br/>Variables and context items can be declared via {{Code|$bindings}} (see [[XQuery Module#xquery:eval|xquery:eval]] for more details)returned. The following {{Code|$options}} parameter contains evaluation optionscan be supplied:* {{Code|cachekeep}}: indicates if Keep the query cached result will be cached or ignored (default: <code>true</code>). If the query result will not be cached, the query id will immediately be discarded error after query execution, tooretrieval.* {{Code|base-uri}}: set [https://www.w3.org/TR/xquery-31/#dt-static-base-uri base-uri property] for the query. This URI will be used when resolving relative URIs by functions such as {{Code|fn:doc}} (default: ''empty string'').|-| '''Errors'''|{{Error|updating|#Errors}} the query contains update operations.|-valign="top"

* {{Code|async:eval("1+3")}} returns a query id, e.g. {{Code|Query-abc}}. The result can be retrieved via a second query in the same BaseX context: {{Code|async:result("Query-abc")}}<br />* The following [[RESTXQ]] function will either return the id of the query thread, which evaluates the query that has been specified in the body result of a POST requestpreviously started job or raise an error:<pre classlang='brush:xquery'>declare %rest:POSTpath("'/result/{$queryid}") %rest:path('/eval') function local:evalresult($queryid) { asyncjob:evalresult($queryid)

==asyncjob:updateremove==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async:update|$query as xs:string|xs:string}}<br /pre>{{Func|asyncjob:update|remove( $query id as xs:string, $bindings options as map(*)|xs? :string= map { }}<br />{{Func|async:update|$query as xs:string, $bindings as map(*), $options as mapempty-sequence(xs:string, xs:string)|xs:string}}<br /pre>|-valign="top"

|Prepares Triggers the supplied cancelation of a job with the specified {{Code|$queryid}} string for asynchronous execution and returns , cancels a query idscheduled job or removes a cached result. Unknown IDs are ignored. The query will be queued as described in All jobs are gracefully stopped; it is up to the article on [[Transaction Management]]process to decide when it is safe to shut down.<br/>See [[#async:evalThe following {{Code|async:eval]] for information on the <code>$bindings</code> and <code>$options</code> arguments.}} can be supplied:|-| '''Errors'''|* {{Error|non-updatingCode|#Errorsservice}} : additionally removes the job from the query does not contain any update operations[[#Services|job services]] list.<br/>|-valign="top"

* <code>asyncjob:updatelist("delete node db)[. != job:opencurrent('db')//text()", map {}, map { 'cache'] ! job: falseremove() }.)</code> returns a query idstops and discards all jobs except for the current one. The text nodes of the database * <code>dbjob:remove(job:current())</code> will be deleted once interrupts the database is available for write accesscurrent job.

==asyncjob:resultwait==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async<pre>job:result|wait( $id as xs:string|item) as empty-sequence()*}}</pre>|-valign="top"

|Returns Waits for the result completion of an asynchronously executed query a job with the specified query {{Code|$id}}:* Results can only be retrieved onceThe function will terminate immediately if the job ID is unknown. After retrievalThis is the case if a future job has not been queued yet, or if the cached result will be ID has already been discardedafter job evaluation.* If the query raised an errorfunction is called with the ID of a queued job, or repeatedly executed job, the error will be raised insteadit may stall and never terminate.|-valign="top"

|{{Error|is-runningself|#Errors}} The current job is addressed.<br/>|} =Listing Jobs= ==job:current== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>job:current() as xs:string</pre>|- valign="top"| '''Summary'''|Returns the ID of the current job.|} ==job:list== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>job:list() as xs:string*</pre>|- valign="top"| '''Summary'''|Returns the query is still IDs of all jobs that are currently registered. The list includes scheduled, queued, running, stopped, and finished jobs with cached results.|- valign="top"| '''Examples'''| <code>job:list()<br/code>returns the same job ID as {{ErrorFunction|unknown|#Errorsjob:current}}if no other job is registered.|}  ==job:list-details== {| width='100%'|- valign="top"| width='120' | '''Signature'''|<pre>job:list-details( $id as xs:string := ()) as element(job)*</pre>|- valign="top"| '''Summary'''|Returns information on all jobs that are currently registered, or on a job with the supplied query specified {{Code|$id }} (or an empty sequence if this job is unknown: not found). The list includes scheduled, queued, running jobs, and cached jobs. A string representation of the job, or its URI, will be returned as a value. The query result may already returned elements have been retrievedadditional attributes:* <code>id</code>: job ID* <code>type</code>: type of the job (command, or query execution may have been stopped, REST, RESTXQ, etc.)* <code>state</code>: current state of the job: <code>scheduled</code>, <code>queued</code>, <code>running</code>, <code>cached</code>* <code>user</code>: user who started the job* <code>duration</code>: evaluation time (included if a job is running or if the result was cached)* <code>start</code>: next start of job (included if a job will be executed repeatedly)* <code>time<br/code>: time when job was registered|-valign="top"

|* The following [[RESTXQ]] function will either return the result of a previously started query or an error:<pre class='brush:xquery'code>declare %rest:path('/result/{$id}') function localjob:resultlist-details($id) { async:result($id)};</precode>* The following query demonstrates how the results of an asynchronously executed query can be returned in a single query. Please note that this is not returns information on the common way how asynchronous query execution is used in practicecurrently running job and possibly others:<pre classlang='brush:xquery'"xml">let $query :<job id="job1" type="XQuery" state="running" user="admin" duration= async:eval('(1 to 10000000)["PT0. = 1]')return (001S"> hofXQUERY job:until( functionlist-details($result) { async:finished($query) }, function($curr) { prof:sleep(10) }, () ), async:result($query))</job>

==asyncjob:finishedbindings==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async<pre>job:finished|bindings( $id as xs:string|xs:boolean}}) as map(*)</pre>|-valign="top"

|Indicates if Returns the evaluation variable bindings of a query an existing job with the specified query {{Code|$id}} has finished. If <code>false</code> is returned, the query is still running. An error will be raised if the query result was not cached or has already been retrieved.|-| '''Errors'''|{{Error|unknown|#Errors}} the supplied query id is unknown: The query result may already no variables have been retrievedbound to this job, or query execution may have been stoppedan empty map is returned.<br/>

==asyncjob:stopfinished==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async<pre>job:stop|finished( $id as xs:string|empty-sequence()}}as xs:boolean</pre>|-valign="top"

|Cancels Indicates if the execution evaluation of a query an already running job with the specified query {{Code|$id}}has finished. As the IDs of finished jobs will usually be discarded, unless caching is enabled, or drops the query result if it has already been executedfunction will also return <code>true</code> for unknown jobs.|-| '''Errors'''|{{Error|unknown|#Errors}} * <code>false</code> indicates that the supplied query id job ID is unknown: The query result may already have been retrievedscheduled, queued, or query execution may have been stoppedcurrently running.* <code>true<br/code>will be returned if the job has either finished, or if the ID is unknown (because the IDs of all finished jobs will not be cached).

==asyncjob:idsservices==

|-valign="top"| width='120' | '''SignaturesSignature'''|{{Func|async:ids||xs<pre>job:stringservices() as element(job)*}}</pre>|-valign="top"

|Returns the ids a list of all queries that are either being executed asynchronously, or jobs that have been executed and the results of which have been cachedpersistently registered as [[#Services|Services]].|-valign="top"| '''ExamplesErrors'''|* <code>async:ids() ! async:stop({{Error|services|#Errors}} Registered services cannot be parsed.)<br/code> stops and invalidates all asynchronous queries and results.

|-valign="top"|{{Code|unexpectedoptions}}| An unexpected error occurred while running a query or function in a separate threadThe specified options are conflicting.|-valign="top"|{{Code|out-of-rangeid}}| The supplied option specified ID is out of rangeinvalid or has already been assigned.|-valign="top"|{{Code|updatingoverflow}}| A Too many queries or query is expected to be non-updating, but it performs updatesresults are queued.|-valign="top"|{{Code|non-updatingrange}}| A query specified time or duration is expected to be updating, but it does not perform updates.|-|{{Code|unknown}}| The supplied query id is unknown or not available anymoreout of range.|-valign="top"|{{Code|is-running}}