-
Notifications
You must be signed in to change notification settings - Fork 32
feat: add decoding as a multi-threaded CLARA engine #1229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
baltzell
wants to merge
45
commits into
development
Choose a base branch
from
decoder-engine
base: development
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
45 commits
Select commit
Hold shift + click to select a range
b625208
add convenience method
baltzell 9ddeb04
remove unused stuff
baltzell 8abc9f9
add decoding engine
baltzell 7fc741b
add it to the clara yaml
baltzell d28d112
use a pool
baltzell c483a65
hmm
baltzell 086a9fe
allow decoder instances to share ConstantsManagers
baltzell 68e6b2f
share ConstantsManagers
baltzell 2fa3ad7
kludge test
baltzell adfab5c
cleanup
baltzell fa21298
try this
baltzell dd5ea34
inherit ConstantsManagers
baltzell e8ad4a1
cleanup
baltzell dfdff95
only check tables if not shared
baltzell 2e24c77
higher ports on macos, cleanup process dpe process
baltzell 7c4d726
remove ineffective pid trap
baltzell 5bfeb49
Revert "higher ports on macos, cleanup process dpe process"
baltzell 74d70ff
Revert "fix job name"
baltzell 3789826
Revert "decouple ubuntu/macos builds to reduce wait"
baltzell 8c939a1
Revert "remove unnecessary reader class"
baltzell 64a202c
Revert "remove example engine"
baltzell 03fdb87
restore reported data type
baltzell c89a8cc
rename class
baltzell 04cb650
undo
baltzell 9f5708b
fix rebase oops
baltzell 1aa6e0a
rename class
baltzell e62ab27
fix rebase oops
baltzell 17c659d
cleanup
baltzell 731ffc2
cleanup
baltzell 0793a15
stf
baltzell 2aacbc1
just events, not per thread
baltzell af5e72f
reduce constants sharing
baltzell be194f3
bugfix
baltzell 1021c02
share one RCDBManager
baltzell 137d3f8
bugfix
baltzell f108b4e
restore
baltzell 236ce28
DecoderEngine: just pass along HIPO events
baltzell 4bd0a9d
add Clas12Reader, no decoding
baltzell bea8f32
dummy initialization value
baltzell 1e3e062
bugfix
baltzell dda612e
split I/O service into separate PR
baltzell f93d36a
split pull requests
baltzell 7e03830
cleanup, avoid class variable
baltzell bc73d3f
remove debugging leftover
baltzell 4f5da7a
revert to hard-coded byte order
baltzell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
113 changes: 113 additions & 0 deletions
113
common-tools/clas-reco/src/main/java/org/jlab/clas/reco/DecoderEngine.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package org.jlab.clas.reco; | ||
|
|
||
| import java.util.Set; | ||
| import java.util.HashSet; | ||
| import java.nio.ByteBuffer; | ||
| import java.nio.ByteOrder; | ||
| import java.util.concurrent.ArrayBlockingQueue; | ||
| import java.util.concurrent.BlockingQueue; | ||
| import org.jlab.clara.base.ClaraUtil; | ||
| import org.jlab.clara.engine.Engine; | ||
| import org.jlab.clara.engine.EngineData; | ||
| import org.jlab.clara.engine.EngineDataType; | ||
| import org.jlab.clara.engine.EngineStatus; | ||
| import org.jlab.detector.decode.CLASDecoder; | ||
| import org.jlab.io.evio.EvioDataEvent; | ||
| import org.jlab.io.hipo.HipoDataEvent; | ||
| import org.jlab.jnp.hipo4.data.SchemaFactory; | ||
| import org.json.JSONObject; | ||
|
|
||
| /** | ||
| * | ||
| * @author baltzell | ||
| */ | ||
| public class DecoderEngine implements Engine { | ||
|
|
||
| static final int POOL_SIZE = 64; | ||
| static final Set<EngineDataType> ED_TYPES = ClaraUtil.buildDataTypes( | ||
| Clas12Types.EVIO,Clas12Types.HIPO,EngineDataType.JSON,EngineDataType.STRING); | ||
|
|
||
| SchemaFactory schema; | ||
| BlockingQueue<CLASDecoder> pool; | ||
| int constantsShared = 16; | ||
|
|
||
| public DecoderEngine() { | ||
| schema = new SchemaFactory(); | ||
| schema.initFromDirectory(System.getenv("CLAS12DIR") + "/etc/bankdefs/hipo4"); | ||
| } | ||
|
|
||
| @Override | ||
| public Set<EngineDataType> getInputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public Set<EngineDataType> getOutputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public EngineData executeGroup(Set<EngineData> set) { return null; } | ||
| @Override | ||
| public Set<String> getStates() { return new HashSet<>(); } | ||
| @Override | ||
| public String getDescription() { return "decoder engine"; } | ||
| @Override | ||
| public String getVersion() { return "1.0"; } | ||
| @Override | ||
| public String getAuthor() { return "baltzell"; } | ||
| @Override | ||
| public void reset() {} | ||
| @Override | ||
| public void destroy() {} | ||
|
|
||
| @Override | ||
| public EngineData configure(EngineData ed) { | ||
| JSONObject json = new JSONObject(ed.getData()); | ||
| pool = new ArrayBlockingQueue<>(POOL_SIZE); | ||
| CLASDecoder d0 = null; | ||
| for (int i=0; i<POOL_SIZE; i++) { | ||
| CLASDecoder d; | ||
| if (i % constantsShared == 0) { | ||
| d0 = new CLASDecoder(); | ||
| if (json.has("variation")) d0.setVariation(json.getString("variation")); | ||
| if (json.has("timestamp")) d0.setVariation(json.getString("timestamp")); | ||
| d = d0; | ||
| } | ||
| else { | ||
| d = new CLASDecoder(d0); | ||
| } | ||
| pool.add(d); | ||
| } | ||
| return ed; | ||
| } | ||
|
|
||
| @Override | ||
| public EngineData execute(EngineData input) { | ||
|
|
||
| EngineData output = input; | ||
|
|
||
| // if it's EVIO, decode it, otherwise just pass it along | ||
| if (input.getMimeType().equals("binary/data-evio")) { | ||
| EvioDataEvent evio; | ||
| try { | ||
| ByteBuffer bb = (ByteBuffer) input.getData(); | ||
| //evio = new EvioDataEvent(bb.array(), bb.order()); | ||
| evio = new EvioDataEvent(bb.array(), ByteOrder.LITTLE_ENDIAN); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error reading input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| HipoDataEvent hipo; | ||
| try { | ||
| CLASDecoder d = pool.take(); | ||
| hipo = new HipoDataEvent(d.getDecodedEvent(evio),schema); | ||
| pool.put(d); | ||
| output.setData("binary/data-hipo", hipo.getHipoEvent()); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error processing input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| } | ||
|
|
||
| return output; | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.