[cybersecurityrtms-3f10-ai] Details
Generated On: 2026-05-22 21:03:15 UTC
TML Solution DAG Parameters' Details: User Chosen Parametets
STEP 1: Get TML Core Params: tml_system_step_1_getparams_dag
User Parameter |
Chosen Value |
solutionname |
cybersecurityrtms-3f10-ai |
solutiontitle |
Entity-Based Real-Time Advanced Cybersecurity Prevention and Monitoring |
solutiondescription |
TML Real-Time Memory of Sliding Time Windows For Advanced Cybersecurity Prevention |
brokerhost |
127.0.0.1 |
brokerport |
9092 |
cloudusername |
None |
ingestdatamethod |
LOCALFILE |
STEP 2: Create Kafka Topics: tml_system_step_2_kafka_createtopic_dag
User Parameter |
Chosen Value |
companyname |
Otics |
myname |
Sebastian |
myemail |
Sebastian.Maurice |
mylocation |
Toronto |
replication |
1 |
numpartitions |
1 |
enabletls |
1 |
microserviceid |
|
raw_data_topic |
iot-raw-data,rtms-stream-mylogs,rtms-stream-mylogs2 |
preprocess_data_topic |
iot-preprocess,iot-preprocess2,rtms-preprocess,attacktopic,rtmstopic,patterntopic |
ml_data_topic |
ml-data |
prediction_data_topic |
prediction-data |
STEP 3: Produce to Kafka Topics
User Parameter |
Chosen Value |
PRODUCETYPE |
LOCALFILE |
inputfile |
/rawdatademo/cisco_network_data.txt |
TOPIC |
iot-raw-data |
PORT |
_39399 |
IDENTIFIER |
TML solution,/rawdatademo/cisco_network_data.txt |
HTTPADDR |
|
FROMHOST |
('seb', '127.0.1.1') |
TOHOST |
0.0.0.0 |
CLIENTPORT |
Not Applicable |
TSS_CLIENTPORT |
Not Applicable |
TML_CLIENTPORT |
Not Applicable |
docfolder |
mylogs,mylogs2 |
doctopic |
rtms-stream-mylogs |
chunks |
3000 |
docingestinterval |
30 |
STEP 4: Preprocesing Data: tml-system-step-4-kafka-preprocess-dag
User Parameter |
Chosen Value |
raw_data_topic |
iot-raw-data,rtms-stream-mylogs,rtms-stream-mylogs2 |
preprocess_data_topic |
iot-preprocess,iot-preprocess2,rtms-preprocess,attacktopic,rtmstopic,patterntopic |
preprocessconditions |
|
delay |
70 |
maxrows |
800 |
array |
0 |
saveasarray |
1 |
topicid |
-999 |
rawdataoutput |
1 |
asynctimeout |
120 |
timedelay |
0 |
preprocesstypes |
anomprob,trend,avg |
pathtotmlattrs |
--pathtotmlattrs-- |
identifier |
RTMS Cybersecurity Prevention |
jsoncriteria |
uid=hostName,filter:allrecords~subtopics=hostName,hostName,hostName~values=inboundpackets,outboundpackets,pingStatus~identifiers=inboundpackets,outboundpackets,pingStatus~datetime=lastUpdated~msgid=~latlong= |
STEP 4a: Preprocesing Data: tml-system-step-4a-kafka-preprocess-dag
User Parameter |
Chosen Value |
raw_data_topic |
rtms-pgpt-ai |
preprocess_data_topic |
rtms-pgpt-ai-mitre |
preprocessconditions |
|
delay |
70 |
maxrows |
50 |
array |
0 |
saveasarray |
1 |
topicid |
-999 |
rawdataoutput |
1 |
asynctimeout |
120 |
timedelay |
0 |
preprocesstypes |
avg |
pathtotmlattrs |
--pathtotmlattrs1-- |
identifier |
Mitre ATTCK |
jsoncriteria |
uid=tactic,filter:allrecords~subtopics=technique,technique,technique~values=FinalAttackScore,FinalPatternScore,RTMSSCORE~identifiers=FinalAttackScore,FinalPatternScore,RTMSSCORE~datetime=TimeStamp~msgid=Entity,PartitionOffsetFound,NumAttackWindowsFound,NumPatternWindowsFound,SearchEntity,rtmsfolder,CurrentRTMSMAXWINDOW~latlong= |
STEP 4b: Preprocesing Data: tml-system-step-4b-kafka-preprocess-dag
User Parameter |
Chosen Value |
raw_data_topic |
--raw_data_topic2-- |
preprocess_data_topic |
--preprocess_data_topic2-- |
preprocessconditions |
--preprocessconditions2-- |
delay |
--delay2-- |
maxrows |
--maxrows2-- |
array |
--array2-- |
saveasarray |
--saveasarray2-- |
topicid |
--topicid2-- |
rawdataoutput |
--rawdataoutput2-- |
asynctimeout |
--asynctimeout2-- |
timedelay |
--timedelay2-- |
preprocesstypes |
--preprocesstypes2-- |
pathtotmlattrs |
--pathtotmlattrs2-- |
identifier |
--identifier2-- |
jsoncriteria |
--jsoncriteria2-- |
STEP 4c: Preprocesing Data: tml-system-step-4c-kafka-preprocess-dag
User Parameter |
Chosen Value |
raw_data_topic |
iot-preprocess |
preprocess_data_topic |
rtms-preprocess |
delay |
70 |
maxrows |
200 |
array |
0 |
saveasarray |
1 |
topicid |
-999 |
rawdataoutput |
1 |
asynctimeout |
120 |
timedelay |
0 |
searchterms |
rgx:p([a-z]+)ch ~~~ |authentication failure,--entity-- password failure ~~~ |unknown--entity-- |
rtmsstream |
rtms-stream-mylogs |
identifier |
RTMS Past Memory of Events |
rememberpastwindows |
500 |
patternwindowthreshold |
30 |
localsearchtermfolder |
|mysearchfile1,|mysearchfile2 |
localsearchtermfolderinterval |
60 |
rtmsscorethreshold |
0.6 |
rtmsscorethresholdtopic |
rtmstopic |
attackscorethreshold |
0.6 |
attackscorethresholdtopic |
attacktopic |
patternscorethreshold |
0.6 |
patternscorethresholdtopic |
patterntopic |
rtmsfoldername |
rtms2 |
rtmsmaxwindows |
1000000 |
RTMS Output Github Link |
STEP 5: Entity Based Machine Learning : tml-system-step-5-kafka-machine-learning-dag
User Parameter |
Chosen Value |
preprocess_data_topic |
iot-preprocess,iot-preprocess2,rtms-preprocess,attacktopic,rtmstopic,patterntopic |
ml_data_topic |
ml-data |
modelruns |
--modelruns-- |
offset |
-1 |
islogistic |
--islogistic-- |
networktimeout |
--networktimeout-- |
modelsearchtuner |
--modelsearchtuner-- |
processlogic |
--processlogic-- |
dependentvariable |
--dependentvariable-- |
independentvariables |
--independentvariables-- |
rollbackoffsets |
--rollbackoffsets-- |
topicid |
-999 |
consumefrom |
rtms-preprocess |
fullpathtotrainingdata |
--fullpathtotrainingdata-- |
transformtype |
--transformtype-- |
sendcoefto |
--sendcoefto-- |
coeftoprocess |
--coeftoprocess-- |
coefsubtopicnames |
--coefsubtopicnames-- |
ML Output Github Link |
STEP 6: Entity Based Predictions: tml-system-step-6-kafka-predictions-dag
User Parameter |
Chosen Value |
preprocess_data_topic |
iot-preprocess,iot-preprocess2,rtms-preprocess,attacktopic,rtmstopic,patterntopic |
ml_prediction_topic |
--ml_prediction_topic-- |
streamstojoin |
--streamstojoin-- |
inputdata |
--inputdata-- |
consumefrom |
--consumefrom2-- |
offset |
-1 |
delay |
70 |
usedeploy |
--usedeploy-- |
networktimeout |
--networktimeout-- |
maxrows |
800 |
topicid |
-999 |
pathtoalgos |
--pathtoalgos-- |
STEP 7: Real-Time Visualization: tml-system-step-7-kafka-visualization-dag
User Parameter |
Chosen Value |
vipervizport |
9689 |
topic |
rtms-pgpt-ai-mitre |
dashboardhtml |
dashboard-rtms-ai-mitre.html |
secure |
1 |
offset |
-1 |
append |
0 |
chip |
amd64 |
rollbackoffset |
400 |
STEP 8: tml_system_step_8_deploy_solution_to_docker_dag
User Parameter |
Chosen Value |
Docker Container |
maadsdocker/cybersecurityrtms-3f10-ai-amd64 (https://hub.docker.com/r/maadsdocker/cybersecurityrtms-3f10-ai-amd64) |
Docker Run Command |
|
STEP 9: tml_system_step_9_privategpt_qdrant_dag
User Parameter |
Chosen Value |
PrivateGPT Container |
maadsdocker/tml-privategpt-with-gpu-nvidia-amd64-v2 |
PrivateGPT Run Command |
docker run -d -p 8001:8001 --net=host --gpus all -v /var/run/docker.sock:/var/run/docker.sock:z --env PORT=8001 --env TSS=1 --env GPU=1 --env COLLECTION=tml-llm-model-v2 --env WEB_CONCURRENCY=2 --env CUDA_VISIBLE_DEVICES=0 --env TOKENIZERS_PARALLELISM=false --env temperature=0.1 --env vectorsearchtype="Manhattan" --env contextwindowsize=4096 --env vectordimension=768 maadsdocker/tml-privategpt-with-gpu-nvidia-amd64-v2 |
Qdrant Container |
qdrant/qdrant |
Qdrant Run Command |
docker run -d -p 6333:6333 -v $(pwd)/qdrant_storage:/qdrant/storage:z qdrant/qdrant |
Consumefrom |
rtms-preprocess |
pgpt_data_topic |
rtms-pgpt-ai |
offset |
-1 |
rollbackoffset |
400 |
topicid |
-999 |
enabletls |
1 |
partition |
-1 |
prompt |
[INST] Are there any errors or suspicious activity in the log messages found? Give a detailed response, and any resolutions that need to be done. Also, Can you give me the MITRE ATT&CK tactic and technique classification for these messages?[/INST] |
context |
This data are from network log files. This log file data have been filtered using the search terms shown in the messages. The filtered messages may indicate potential suspicious log entries that could indicate a cyber attack. |
jsonkeytogather |
SearchTextFound |
keyattribute |
|
keyprocesstype |
|
vectordbcollectionname |
tml-llm-model-v2 |
concurrency |
2 |
CUDA_VISIBLE_DEVICES |
0 |
pgpthost |
|
pgptport |
8001 |
hyperbatch |
0 |
docfolder |
--docfolder-- |
docfolderingestinterval |
900 |
useidentifierinprompt |
1 |
searchterms |
--searchterms-- |
streamall |
1 |
temperature |
0.1 |
vectorsearchtype |
Manhattan |
llm |
|
embedding |
|
vectorsize |
|
contextwindowsize |
4096 |
vectordimension |
768 |
mitrejson |
/rawdata/mitre.json |
STEP 9b: tml_system_step_9b_agenticai_dag
User Parameter |
Chosen Value |
rollbackoffset |
--agenticai-rollbackoffset-- |
ollama-model |
--agenticai-ollama-model-- |
deletevectordbcount |
--agenticai-deletevectordbcount-- |
vectordbpath |
--agenticai-vectordbpath-- |
temperature |
--agenticai-temperature-- |
topicid |
--agenticai-topicid-- |
enabletls |
--agenticai-enabletls-- |
partition |
--agenticai-partition-- |
vectordbcollectionname |
--agenticai-vectordbcollectionname-- |
ollamacontainername |
--agenticai-ollamacontainername-- |
mainip |
--agenticai-mainip-- |
mainport |
--agenticai-mainport-- |
embedding |
--agenticai-embedding-- |
agenttopic |
--agenticai-agenttopic-- |
agents_topic_prompt |
--agenticai-agents_topic_prompt-- |
teamlead_topic |
--agenticai-teamlead_topic-- |
teamleadprompt |
--agenticai-teamleadprompt-- |
supervisor_topic |
--agenticai-supervisor_topic-- |
supervisorprompt |
--agenticai-supervisorprompt-- |
agenttoolfunctions |
--agenticai-agenttoolfunctions-- |
agent_team_supervisor_topic |
--agenticai-agent_team_supervisor_topic-- |
concurrency |
--agenticai-concurrency-- |
CUDA_VISIBLE_DEVICES |
--agenticai-cuda-- |
contextwindow |
--agenticai-contextwindow-- |
localmodelsfolder |
--agenticai-localmodelsfolder-- |
STEP 10: tml_system_step_10_documentation_dag
User Parameter |
Chosen Value |
Solution Documentation URL |