Date: Sun, 12 Aug 2007 09:45:41 -0400
Reply-To: "Richard A. DeVenezia" <rdevenezia@WILDBLUE.NET>
Sender: "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From: "Richard A. DeVenezia" <rdevenezia@WILDBLUE.NET>
Subject: Re: Streaming Tick data from a trading platform to a SAS app
On Mon, 6 Aug 2007 13:39:44 -0000, maskiepop <maskiepop@GMAIL.COM> wrote:
>On Aug 6, 5:56 pm, ben.pow...@CLA.CO.UK wrote:
>> If you can identify a source of data and post the format if necessary then
>> importing that into sas should be straightforward. You can review the
>> output in a variety of ways - ODS html would be the first that springs to
>> mind.
>>
>> HTH.
>
>Hi ben,
>
>Thanks for replying to my post. The data, when it reaches the trading
>platform, from the exchange (CME), will be of two types:
>
>A: trades
>
>o timestamp - UTC
>o price
>o quantity
>o transaction type: "trade"
>
>B: quotes
>
>o timestamp - UTC
>o ask
>o bid
>o asksize
>o bidsize
>o transaction type: "quote"
>
>I realize I will have to mess around with the API of the trading
>platform, whatever it is. I'd be happy if someone had done this
>already, on a particular platform, and on the same data - the emini
>S&P 500 from CME. In which case, I'm prepared to get the same
>platform, and follow his/her lead. Sounds rather silly; could lead to
>problems, I know, but I have very little experience with C#, C++,
>Java, and the Windows platform. Plus I'd rather be working on my SAS
>app and get it's trading logic right rather than spend time building
>the infrastructure. Unless I have to of course.
>
>At any rate, we all have to assume that I can put enough code on the
>trading platform to get the data in the format I want; then my problem
>is sending the data from the trading platform to my SAS app. I was
>thinking of building a named pipe feature inside the trading platform,
>and piping the data from the trading platform to the SAS app, one tick
>at a time, and as they arrive from the exchange. The technicians I
>consulted, didn't like the idea, didn't say why, but simply said, "We
>can't support that.". If anyone understands the issues surrounding the
>use of named pipes in a trading platform to send data to an external
>app, I'd be happy to listen and be less informed.
>
>Another solution came from someone who passed tick data from a trading
>platform to an external app that he wrote in C++. He used WM_COPY to
>send the data, as strings, to his app. I understand this is Windows
>messaging. I am looking at the code, and would like to see if I can
>get a simple SAS code to receive the message. It will mean getting
>"simple SAS" to interface with an external DLL. The part I haven't
>figured out is how "simple SAS" can be made to sit there and listen,
>then react when it receives an incoming message. (In the case of the C+
>+ code, it simply looped around the receiving logic, all the while
>testing for the existence of an incoming message.) What are we talking
>about here? AF? SCL? Is this not doable in batch? If there is a batch
>solution, I'd prefer that. But then again, beggars can't be choosers.
>
>Note that importing the data, as in importing csv data to excel, is
>not the way to go, as indicated above.
>
>Thanks.
You can create a stream adapter that reads from some streaming source and
writes to a SAS receiver listening at some socket port.
The receiver runs continuously in batch and creates small sequentially named
tables containing new data it has received.
The analytic component gathers the small tables together and appends them to
a base table on which the analytics will occur. This is not a realtime
system, but affords you the ability to do "close to live" snapshot analytics.
For a real-time 'live view' system, you are probably looking at a SAS/AF
application or some other application framework which can connect itself to
the stream source.
Consider this example:
Folders
demo
\-- source
\-- sink
\-- analyst
\--common
demo\source\source.sas is a 'stream adapter emulator'. Replace it with your
real stream adapater (it will read from a stream and write to a socket).
demo\sink\sink.sas is a tcp/ip server that listens for input from the stream
adapter. It accretes received data into small tables that are created
dynamically from within an infinite data _null_ using the hash object.
demo\analyst\analysts.sas is some program that is analyzing 'streamed' data.
It stacks all new accretions into a clump which is appended to some base
table that is then analyzed.
demo\common is a library shared between programs in sink and analyst.
Consider
-----------------------
demo\start-simulation.bat
-----------------------
del /q common\*
cd source
start "source - stream adapter emulation" c:\opt\sas\v913\sas.exe -sasuser
.\sasuser -noterminal -batch -nosleepwindow -sysin source.sas
cd ..\sink
start "sink" c:\opt\sas\v913\sas.exe -sasuser
.\sasuser -noterminal -batch -sysin sink.sas
cd ..\analyst
c:\opt\sas\v913\sas.exe -sasuser .\sasuser -noterminal -batch -sysin
analyst.sas -log analyst-1.log -nosleepwindow
c:\opt\sas\v913\sas.exe -sasuser .\sasuser -noterminal -batch -sysin
analyst.sas -log analyst-2.log -nosleepwindow
c:\opt\sas\v913\sas.exe -sasuser .\sasuser -noterminal -batch -sysin
analyst.sas -log analyst-3.log -nosleepwindow
-----------------------
-----------------------
demo\source\source.sas
-----------------------
filename drip socket 'localhost:5000';
data _null_;
file drip;
do _n_ = 1 by 1;
counter+1;
datetime = datetime();
id = ceil(100*ranuni(1234));
price = 10 + sin(counter/6) + ranuni(1234)/100;
putlog _n_ datetime 16.4 id 4. +1 price 9.4;
put _n_ datetime 16.4 id 4. +1 price 9.4;
rc = sleep(1+20*ranuni(1234),0.001);
* demo guard;
if counter > 400 then do;
put '-1 -1 -1 -1';
stop;
end;
end;
run;
-----------------------
-----------------------
demo\sink\sink.sas
-----------------------
filename collect socket 'localhost:5000' server reconn=3;
libname common '..\common';
%let every = 10;
data _null_;
declare hash accretion;
if mod(_n_,&every) = 1 then do;
accretion = _new_ hash(ordered:'y');
accretion.definekey('tickdt');
accretion.definedata('rowid', 'tickdt', 'id', 'price');
accretion.definedone();
end;
infile collect eov=v;
input rowid tickdt id price;
format rowid 8. tickdt 16.4 id 4. price 8.4;
putlog rowid 6. +1 tickdt 16.4 id 4. +1 price 9.4;
if tickdt ne -1 then
accretion.add();
if mod(_n_,&every) = 0 or tickdt=-1 then do;
counter + 1;
accretion.output(dataset:'common.accretion_'||put(counter,z8.));
if tickdt = -1 then stop;
accretion.delete();
end;
run;
-----------------------
-----------------------
demo\analyst\analyst.sas
-----------------------
data _null_;
call sleep (2,1);
run;
libname common '..\common';
proc sql noprint;
select
cats(libname,'.',memname)
into
:accretions separated by ' '
from
dictionary.tables
where
libname EQ 'COMMON'
& memname EQT 'ACCRETION_'
order by
memname
;
quit;
data _null_;
if &SQLOBS. = 0 then
call execute ('endsas;');
run;
data clump;
set &accretions.;
run;
proc append
base=common.tickdata
new=clump
;
run;
proc sql;
drop table %sysfunc(translate(&accretions.,%str(,),%str( )));
quit;
* do whatever analytic is desired;
-----------------------
Richard A. DeVenezia
http://www.devenezia.com/
