
1. stopping the workflows
-------------------------
Stopping does not stop. Kepler is stucked in wrapping up state for very long. 
The wrap up procedure must have a bug.

2. Timesteps [constraint on simulation emulation]
-------------------------------------------------
The files with different timesteps must have different times when listing with 'ls -l'.
Otherwise they come mixed and not in a row. I.e. at least a minute difference!

A big simulation outputs files slowly but what if a simulation writes two steps in a minute?

3. ProcessFile does not check for output file size and date
-----------------------------------------------------------
The ProcessFile creates the output file name from specification and assembles the output
record from the output name and the size and date of the input file, which is clearly 
wrong. To sound working, ProcessFile should get the size and date of the produced output
file and pack that into the output record.
Due to this bug, archiving gets the sizes of the original transferred files, not the converted 
ones.


5. Archiver name
----------------
The first archive name will look like:  xgc-5-2550 instead of xgc-2500-2550 because the first file
is xgc.mesh.h5 which has no timestep. I have no idea what could be a correct solution. 

6. Running script and exiting from Kepler aborts the script
-----------------------------------------------------------
Which is generally okay, but this way it is dangerous to run the simulation for 22 minutes,
meanwhile the user builds the other workflow.
Nohup does not help alone, & cannot be applied because then the workflow finishes 
immediately.


=======
 Fixed
=======

1. -1 exit code at ssh commands [Showstopper]
---------------------------------------------
Interestingly, bugs do not come out at testing phase of development, only before 
hands-on tutorials :-(
Sometimes ssh commands return -1 even with correct execution. This must be something
with multiple command execution using the same session. 
See org.kepler.ssh.SshExec class, executeCmd() method.

Have no idea what is going on. 

fix: org.kepler.ssh.SshExec. Exit code must be asked only after the channel is closed.


2. File locations [Inconvenient]
--------------------------------
the log file will be created in this directory (it uses FileParameter) 
the ckpt files will be created in the directory where Kepler starts (Java user.dir)
the job cmd file is also taken relatively to user.dir 

This should be consistent (or use absolute path names)

fix: org.kepler.actor.MappedLogger is similar to Logger and this actor is used
for checkpointing. There is no job submission at the hands-on...

3. ssh password authentication
------------------------------
If the password is wrong, the authentication fails. But no new password is asked. 
At rerun the wrong password is automatically reused, so we have to restart the
whole Kepler.

fix: org.kepler.ssh.SshSession now reuses a pwd only if it was used successfully.

4. Archiver rerun
-------------------
If all files have been processed in the first run, a rerun will produce a arch--1--1.tar
(timesteps -1..-1). This should be fixed in ArchiveCounter.

fix: special case handled: if stop file arrives when list is empty then the list is not emitted.


5. Idle ssh sessions break
--------------------------
After long idle periods an opened ssh session may be closed down on the remote site.
This is not realized on this side until a new channel is tried to be opened. 
The "sesson is down" exception is now catched, the session is closed and reopened, and 
then the channel is retried.
This is fixed only for SshExec:executeCmd() and not for copyTo() and copyFrom(), which should 
be fixed as well. But for the hands-on, this is fine.

6. Archiver workflow bug
------------------------
The current archiver SDF/DDF combo works if the transfer of the list from local to remote machine
succeeds. Otherwise, token rate problems raise Ptolemy exception. The rest of the workflow still runs
but the archiver part fails. 

fix: new workflow done

