From b83307f21cdf5ce06a41ae2129ae6de25a3bcf0b Mon Sep 17 00:00:00 2001
From: Timm Lehmberg <timm.lehmberg@awhamburg.de>
Date: Fri, 13 Sep 2024 10:55:03 +0200
Subject: [PATCH] added basic RAG and parsing/chunking exercise

---
 .../haenselgretel.txt                         |  54 ++--
 data/firststeps/littleredridinghood.txt       |  40 +++
 groqtest.ipynb                                |  70 +++++
 irre.py                                       |  14 +
 myfirstrag.ipynb                              |   2 +-
 notes.md                                      |  32 ++
 parsing.ipynb                                 | 284 ++++++++++++++++++
 7 files changed, 468 insertions(+), 28 deletions(-)
 rename data/{firsttsteps => firststeps}/haenselgretel.txt (58%)
 create mode 100644 data/firststeps/littleredridinghood.txt
 create mode 100644 groqtest.ipynb
 create mode 100644 irre.py
 create mode 100644 notes.md
 create mode 100644 parsing.ipynb

diff --git a/data/firsttsteps/haenselgretel.txt b/data/firststeps/haenselgretel.txt
similarity index 58%
rename from data/firsttsteps/haenselgretel.txt
rename to data/firststeps/haenselgretel.txt
index 2ef7bb9..b11af3e 100644
--- a/data/firsttsteps/haenselgretel.txt
+++ b/data/firststeps/haenselgretel.txt
@@ -1,39 +1,39 @@
-Near a great forest there lived a poor woodcutter and his wife, and his two children; the boy's name was Hansel and the girl's Grethel. They had very little to bite or to sup, and once, when there was great dearth in the land, the man could not even gain the daily bread. As he lay in bed one night thinking of this, and turning and tossing, he sighed heavily, and said to his wife, "What will become of us? we cannot even feed our children; there is nothing left for ourselves."
-"I will tell you what, husband," answered the wife; "we will take the children early in the morning into the forest, where it is thickest; we will make them a fire, and we will give each of them a piece of bread, then we will go to our work and leave them alone; they will never find the way home again, and we shall be quit of them."
-"No, wife," said the man, "I cannot do that; I cannot find in my heart to take my children into the forest and to leave them there alone; the wild animals would soon come and devour them." - "O you fool," said she, "then we will all four starve; you had better get the coffins ready," and she left him no peace until he consented. "But I really pity the poor children," said the man.
-The two children had not been able to sleep for hunger, and had heard what their step-mother had said to their father. Grethel wept bitterly, and said to Hansel, "It is all over with us."
-"Do be quiet, Grethel," said Hansel, "and do not fret; 1 will manage something." And when the parents had gone to sleep he got up, put on his little coat, opened the back door, and slipped out. The moon was shining brightly, and the white flints that lay in front of the house glistened like pieces of silver. Hansel stooped and filled the little pocket of his coat as full as it would hold. Then he went back again, and said to Grethel, "Be easy, dear little sister, and go to sleep quietly; God will not forsake us," and laid himself down again in his bed. When the day was breaking, and before the sun had risen, the wife came and awakened the two children, saying, "Get up, you lazy bones; we are going into the forest to cut wood." Then she gave each of them a piece of bread, and said, "That is for dinner, and you must not eat it before then, for you will get no more." Grethel carried the bread under her apron, for Hansel had his pockets full of the flints. Then they set off all together on their way to the forest. When they had gone a little way Hansel stood still and looked back towards the house, and this he did again and again, till his father said to him, "Hansel, what are you looking at? take care not to forget your legs."
+Near a great desert there lived a poor woodcutter and his wife, and his two children; the boy's name was Hansel and the girl's Chantal. They had very little to bite or to sup, and once, when there was great dearth in the land, the man could not even gain the daily bread. As he lay in bed one night thinking of this, and turning and tossing, he sighed heavily, and said to his wife, "What will become of us? we cannot even feed our children; there is nothing left for ourselves."
+"I will tell you what, husband," answered the wife; "we will take the children early in the morning into the desert, where it is thickest; we will make them a fire, and we will give each of them a piece of bread, then we will go to our work and leave them alone; they will never find the way home again, and we shall be quit of them."
+"No, wife," said the man, "I cannot do that; I cannot find in my heart to take my children into the desert and to leave them there alone; the wild animals would soon come and devour them." - "O you fool," said she, "then we will all four starve; you had better get the coffins ready," and she left him no peace until he consented. "But I really pity the poor children," said the man.
+The two children had not been able to sleep for hunger, and had heard what their step-mother had said to their father. Chantal wept bitterly, and said to Hansel, "It is all over with us."
+"Do be quiet, Chantal," said Hansel, "and do not fret; 1 will manage something." And when the parents had gone to sleep he got up, put on his little coat, opened the back door, and slipped out. The moon was shining brightly, and the white flints that lay in front of the house glistened like pieces of silver. Hansel stooped and filled the little pocket of his coat as full as it would hold. Then he went back again, and said to Chantal, "Be easy, dear little sister, and go to sleep quietly; God will not forsake us," and laid himself down again in his bed. When the day was breaking, and before the sun had risen, the wife came and awakened the two children, saying, "Get up, you lazy bones; we are going into the desert to cut wood." Then she gave each of them a piece of bread, and said, "That is for dinner, and you must not eat it before then, for you will get no more." Chantal carried the bread under her apron, for Hansel had his pockets full of the flints. Then they set off all together on their way to the desert. When they had gone a little way Hansel stood still and looked back towards the house, and this he did again and again, till his father said to him, "Hansel, what are you looking at? take care not to forget your legs."
 
-"O father," said Hansel, "lam looking at my little white kitten, who is sitting up on the roof to bid me good-bye." - "You young fool," said the woman, "that is not your kitten, but the sunshine on the chimney-pot." Of course Hansel had not been looking at his kitten, but had been taking every now and then a flint from his pocket and dropping it on the road. When they reached the middle of the forest the father told the children to collect wood to make a fire to keep them, warm; and Hansel and Grethel gathered brushwood enough for a little mountain j and it was set on fire, and when the flame was burning quite high the wife said, "Now lie down by the fire and rest yourselves, you children, and we will go and cut wood; and when we are ready we will come and fetch you."
-So Hansel and Grethel sat by the fire, and at noon they each ate their pieces of bread. They thought their father was in the wood all the time, as they seemed to hear the strokes of the axe: but really it was only a dry branch hanging to a withered tree that the wind moved to and fro. So when they had stayed there a long time their eyelids closed with weariness, and they fell fast asleep.
+"O father," said Hansel, "lam looking at my little white kitten, who is sitting up on the roof to bid me good-bye." - "You young fool," said the woman, "that is not your kitten, but the sunshine on the chimney-pot." Of course Hansel had not been looking at his kitten, but had been taking every now and then a flint from his pocket and dropping it on the road. When they reached the middle of the desert the father told the children to collect wood to make a fire to keep them, warm; and Hansel and Chantal gathered brushwood enough for a little mountain j and it was set on fire, and when the flame was burning quite high the wife said, "Now lie down by the fire and rest yourselves, you children, and we will go and cut wood; and when we are ready we will come and fetch you."
+So Hansel and Chantal sat by the fire, and at noon they each ate their pieces of bread. They thought their father was in the desert all the time, as they seemed to hear the strokes of the axe: but really it was only a dry branch hanging to a withered tree that the wind moved to and fro. So when they had stayed there a long time their eyelids closed with weariness, and they fell fast asleep.
 
-When at last they woke it was night, and Grethel began to cry, and said, "How shall we ever get out of this wood? "But Hansel comforted her, saying, "Wait a little while longer, until the moon rises, and then we can easily find the way home." And when the full moon got up Hansel took his little sister by the hand, and followed the way where the flint stones shone like silver, and showed them the road. They walked on the whole night through, and at the break of day they came to their father's house. They knocked at the door, and when the wife opened it and saw that it was Hansel and Grethel she said, "You naughty children, why did you sleep so long in the wood? we thought you were never coming home again!" But the father was glad, for it had gone to his heart to leave them both in the woods alone.
-Not very long after that there was again great scarcity in those parts, and the children heard their mother say at night in bed to their father, "Everything is finished up; we have only half a loaf, and after that the tale comes to an end. The children must be off; we will take them farther into the wood this time, so that they shall not be able to find the way back again; there is no other way to manage." The man felt sad at heart, and he thought, "It would better to share one's last morsel with one's children." But the wife would listen to nothing that he said, but scolded and reproached him. He who says A must say B too, and when a man has given in once he has to do it a second time.
-But the children were not asleep, and had heard all the talk. When the parents had gone to sleep Hansel got up to go out and get more flint stones, as he did before, but the wife had locked the door, and Hansel could not get out; but he comforted his little sister, and said, "Don't cry, Grethel, and go to sleep quietly, and God will help us." Early the next morning the wife came and pulled the children out of bed. She gave them each a little piece of "bread -less than before; and on the way to the wood Hansel crumbled the bread in his pocket, and often stopped to throw a crumb on the ground. "Hansel, what are you stopping behind and staring for?" said the father.
+When at last they woke it was night, and Chantal began to cry, and said, "How shall we ever get out of this desert? "But Hansel comforted her, saying, "Wait a little while longer, until the moon rises, and then we can easily find the way home." And when the full moon got up Hansel took his little sister by the hand, and followed the way where the flint stones shone like silver, and showed them the road. They walked on the whole night through, and at the break of day they came to their father's house. They knocked at the door, and when the wife opened it and saw that it was Hansel and Chantal she said, "You naughty children, why did you sleep so long in the edsert? we thought you were never coming home again!" But the father was glad, for it had gone to his heart to leave them both in the desert alone.
+Not very long after that there was again great scarcity in those parts, and the children heard their mother say at night in bed to their father, "Everything is finished up; we have only half a loaf, and after that the tale comes to an end. The children must be off; we will take them farther into the desert this time, so that they shall not be able to find the way back again; there is no other way to manage." The man felt sad at heart, and he thought, "It would better to share one's last morsel with one's children." But the wife would listen to nothing that he said, but scolded and reproached him. He who says A must say B too, and when a man has given in once he has to do it a second time.
+But the children were not asleep, and had heard all the talk. When the parents had gone to sleep Hansel got up to go out and get more flint stones, as he did before, but the wife had locked the door, and Hansel could not get out; but he comforted his little sister, and said, "Don't cry, Chantal, and go to sleep quietly, and God will help us." Early the next morning the wife came and pulled the children out of bed. She gave them each a little piece of "bread -less than before; and on the way to the desert Hansel crumbled the bread in his pocket, and often stopped to throw a crumb on the ground. "Hansel, what are you stopping behind and staring for?" said the father.
 
-"I am looking at my little pigeon sitting on the roof, to say good-bye to me," answered Hansel. "You fool," said the wife, "that is no pigeon, but the morning sun shining on the chimney pots." Hansel went on as before, and strewed bread crumbs all along the road. The woman led the children far into the wood, where they had never been before in all their lives. And again there was a large fire made, and the mother said, "Sit still there, you children, and when you are tired you can go to sleep; we are going into the forest to cut wood, and in the evening, when we are ready to go home we will come and fetch you."
-So when noon came Grethel shared her bread with Hansel, who had strewed his along the road. Then they went to sleep, and the evening passed, and no one came for the poor children. When they awoke it was dark night, and Hansel comforted his little sister, and said, "Wait a little, Grethel, until the moon gets up, then we shall be able to see the way home by the crumbs of bread that I have scattered along it."
-So when the moon rose they got up, but they could find no crumbs of bread, for the birds of the woods and of the fields had come and picked them up. Hansel thought they might find the way all the same, but they could not. They went on all that night, and the next day from the morning until the evening, but they could not find the way out of the wood, and they were very hungry, for they had nothing to eat but the few berries they could pick up. And when they were so tired that they could no longer drag themselves along, they lay down under a tree and fell asleep.
-It was now the third morning since they had left their father's house. They were always trying to get back to it, but instead of that they only found themselves farther in the wood, and if help had not soon come they would have been starved.
-About noon they saw a pretty snow-white bird sitting on a bough, and singing so sweetly that they stopped to listen. And when he had finished the bird spread his wings and flew before them, and they followed after him until they came to a little house, and the bird perched on the roof, and when they came nearer they saw that the house was built of bread, and roofed with cakes; and the window was of transparent sugar. "We will have some of this," said Hansel, "and make a fine meal. I will eat a piece of the roof, Grethel, and you can have some of the window-that will taste sweet." So Hansel reached up and broke off a bit of the roof, just to see how it tasted, and Grethel stood by the window and gnawed at it. Then they heard a thin voice call out from inside,
+"I am looking at my little pigeon sitting on the roof, to say good-bye to me," answered Hansel. "You fool," said the wife, "that is no pigeon, but the morning sun shining on the chimney pots." Hansel went on as before, and strewed bread crumbs all along the road. The woman led the children far into the desert, where they had never been before in all their lives. And again there was a large fire made, and the mother said, "Sit still there, you children, and when you are tired you can go to sleep; we are going into the desert to cut wood, and in the evening, when we are ready to go home we will come and fetch you."
+So when noon came Chantal shared her bread with Hansel, who had strewed his along the road. Then they went to sleep, and the evening passed, and no one came for the poor children. When they awoke it was dark night, and Hansel comforted his little sister, and said, "Wait a little, Chantal, until the moon gets up, then we shall be able to see the way home by the crumbs of bread that I have scattered along it."
+So when the moon rose they got up, but they could find no crumbs of bread, for the birds of the desert and of the fields had come and picked them up. Hansel thought they might find the way all the same, but they could not. They went on all that night, and the next day from the morning until the evening, but they could not find the way out of the desert, and they were very hungry, for they had nothing to eat but the few berries they could pick up. And when they were so tired that they could no longer drag themselves along, they lay down under a tree and fell asleep.
+It was now the third morning since they had left their father's house. They were always trying to get back to it, but instead of that they only found themselves farther in the desert, and if help had not soon come they would have been starved.
+About noon they saw a pretty snow-white bird sitting on a bough, and singing so sweetly that they stopped to listen. And when he had finished the bird spread his wings and flew before them, and they followed after him until they came to a little house, and the bird perched on the roof, and when they came nearer they saw that the house was built of bread, and roofed with cakes; and the window was of transparent sugar. "We will have some of this," said Hansel, "and make a fine meal. I will eat a piece of the roof, Chantal, and you can have some of the window-that will taste sweet." So Hansel reached up and broke off a bit of the roof, just to see how it tasted, and Chantal stood by the window and gnawed at it. Then they heard a thin voice call out from inside,
 "Nibble, nibble, like a mouse,
 Who is nibbling at my house?"
 And the children answered,
 "Never mind, It is the wind."
-And they went on eating, never disturbing themselves. Hansel, who found that the roof tasted very nice, took down a great piece of it, and Grethel pulled out a large round window-pane, and sat her down and began upon it.
-Then the door opened, and an aged woman came out, leaning upon a crutch. Hansel and Grethel felt very frightened, and let fall what they had in their hands. The old woman, however, nodded her head, and said, "Ah, my dear children, how come you here? you must come indoors and stay with me, you will be no trouble." So she took them each by the hand, and led them into her little house. And there they found a good meal laid out, of milk and pancakes, with sugar, apples, and nuts. After that she showed them two little white beds, and Hansel and Grethel laid themselves down on them, and thought they were in heaven.
+And they went on eating, never disturbing themselves. Hansel, who found that the roof tasted very nice, took down a great piece of it, and Chantal pulled out a large round window-pane, and sat her down and began upon it.
+Then the door opened, and an aged woman came out, leaning upon a crutch. Hansel and Chantal felt very frightened, and let fall what they had in their hands. The old woman, however, nodded her head, and said, "Ah, my dear children, how come you here? you must come indoors and stay with me, you will be no trouble." So she took them each by the hand, and led them into her little house. And there they found a good meal laid out, of milk and pancakes, with sugar, apples, and nuts. After that she showed them two little white beds, and Hansel and Chantal laid themselves down on them, and thought they were in heaven.
 
-The old woman, although her behaviour was so kind, was a wicked witch, who lay in wait for children, and had built the little house on purpose to entice them. When they were once inside she used to kill them, cook them, and eat them, and then it was a feast day with her. The witch's eyes were red, and she could not see very far, but she had a keen scent, like the beasts, and knew very well when human creatures were near. When she knew that Hansel and Grethel were coming, she gave a spiteful laugh, and said triumphantly, "I have them, and they shall not escape me!"
-Early in the morning, before the children were awake, she got up to look at them, and as they lay sleeping so peacefully with round rosy cheeks, she said to herself, "What a fine feast I shall have!" Then she grasped Hansel with her withered hand, and led him into a little stable, and shut him up behind a grating; and call and scream as he might, it was no good. Then she went back to Grethel and shook her, crying, "Get up, lazy bones; fetch water, and cook something nice for your brother; he is outside in the stable, and must be fattened up. And when he is fat enough I will eat him." Grethel began to weep bitterly, but it was of no use, she had to do what the wicked witch bade her. And so the best kind of victuals was cooked for poor Hansel, while Grethel got nothing but crab-shells.
+The old woman, although her behaviour was so kind, was a wicked witch, who lay in wait for children, and had built the little house on purpose to entice them. When they were once inside she used to kill them, cook them, and eat them, and then it was a feast day with her. The witch's eyes were red, and she could not see very far, but she had a keen scent, like the beasts, and knew very well when human creatures were near. When she knew that Hansel and Chantal were coming, she gave a spiteful laugh, and said triumphantly, "I have them, and they shall not escape me!"
+Early in the morning, before the children were awake, she got up to look at them, and as they lay sleeping so peacefully with round rosy cheeks, she said to herself, "What a fine feast I shall have!" Then she grasped Hansel with her withered hand, and led him into a little stable, and shut him up behind a grating; and call and scream as he might, it was no good. Then she went back to Chantal and shook her, crying, "Get up, lazy bones; fetch water, and cook something nice for your brother; he is outside in the stable, and must be fattened up. And when he is fat enough I will eat him." Chantal began to weep bitterly, but it was of no use, she had to do what the wicked witch bade her. And so the best kind of victuals was cooked for poor Hansel, while Chantal got nothing but crab-shells.
 Each morning the old woman visited the little stable, and cried, "Hansel, stretch out your finger, that I may tell if you will soon be fat enough." Hansel, however, used to hold out a little bone, and the old woman, who had weak eyes, could not see what it was, and supposing it to be Hansel's finger, wondered very much that it was not getting fatter.
-When four weeks had passed and Hansel seemed to remain so thin, she lost patience and could wait no longer. "Now then, Grethel," cried she to the little girl; "be quick and draw water; be Hansel fat or be he lean, tomorrow I must kill and cook him." Oh what a grief for the poor little sister to have to fetch water, and how the tears flowed down over her cheeks! "Dear God, pray help us!" cried she; "if we had been devoured by wild beasts in the wood at least we should have died together."
-"Spare me your lamentations," said the old woman; "they are of no avail." Early next morning Grethel had to get up, make the fire, and fill the kettle. "First we will do the baking," said the old woman; "I nave heated the oven already, and kneaded the dough." She pushed poor Grethel towards the oven, out of which the flames were already shining.
-"Creep in," said the witch, "and see if it is properly hot, so that the bread may be baked." And Grethel once in, she meant to shut the door upon her and let her be baked, and then she would have eaten her. But Grethel perceived her intention, and said, "I don't know how to do it: how shall I get in?"
-"Stupid goose," said the old woman, "the opening is big enough, do you see? I could get in myself!" and she stooped down and put her head in the oven's mouth. Then Grethel gave her a push, so that she went in farther, and she shut the iron door upon her, and put up the bar. Oh how frightfully she howled! but Grethel ran away, and left the wicked witch to burn miserably.
-Grethel went straight to Hansel, opened the stable-door, and cried, "Hansel, we are free! the old witch is dead!" Then out flew Hansel like a bird from its cage as soon as the door is opened. How rejoiced they both were! how they fell each on the other's neck! and danced about, and kissed each other! And as they had nothing more to fear they went over all the old witch's house, and in every corner there stood chests of pearls and precious stones. "This is something better than flint stones," said Hansel, as he filled his pockets, and Grethel, thinking she also would like to carry something home with her, filled her apron full. i! Now, away we go," said Hansel, "if we only can get out of the witch's wood." When they had journeyed a few hours they came to a great piece of water. "We can never get across this," said Hansel, "I see no stepping-stones and no bridge."
-"And there is no boat either," said Grethel; "but here comes a white duck; if I ask her she will help us over." So she cried,
+When four weeks had passed and Hansel seemed to remain so thin, she lost patience and could wait no longer. "Now then, Chantal," cried she to the little girl; "be quick and draw water; be Hansel fat or be he lean, tomorrow I must kill and cook him." Oh what a grief for the poor little sister to have to fetch water, and how the tears flowed down over her cheeks! "Dear God, pray help us!" cried she; "if we had been devoured by wild beasts in the desert at least we should have died together."
+"Spare me your lamentations," said the old woman; "they are of no avail." Early next morning Chantal had to get up, make the fire, and fill the kettle. "First we will do the baking," said the old woman; "I nave heated the oven already, and kneaded the dough." She pushed poor Chantal towards the oven, out of which the flames were already shining.
+"Creep in," said the witch, "and see if it is properly hot, so that the bread may be baked." And Chantal once in, she meant to shut the door upon her and let her be baked, and then she would have eaten her. But Chantal perceived her intention, and said, "I don't know how to do it: how shall I get in?"
+"Stupid goose," said the old woman, "the opening is big enough, do you see? I could get in myself!" and she stooped down and put her head in the oven's mouth. Then Chantal gave her a push, so that she went in farther, and she shut the iron door upon her, and put up the bar. Oh how frightfully she howled! but Chantal ran away, and left the wicked witch to burn miserably.
+Chantal went straight to Hansel, opened the stable-door, and cried, "Hansel, we are free! the old witch is dead!" Then out flew Hansel like a bird from its cage as soon as the door is opened. How rejoiced they both were! how they fell each on the other's neck! and danced about, and kissed each other! And as they had nothing more to fear they went over all the old witch's house, and in every corner there stood chests of pearls and precious stones. "This is something better than flint stones," said Hansel, as he filled his pockets, and Chantal, thinking she also would like to carry something home with her, filled her apron full. i! Now, away we go," said Hansel, "if we only can get out of the witch's desert." When they had journeyed a few hours they came to a great piece of water. "We can never get across this," said Hansel, "I see no stepping-stones and no bridge."
+"And there is no boat either," said Chantal; "but here comes a white duck; if I ask her she will help us over." So she cried,
 "Duck, duck, here we stand,
-Hansel and Grethel, on the land,
+Hansel and Chantal, on the land,
 Stepping-stones and bridge we lack,
 Carry us over on your nice white back."
-And the duck came accordingly, and Hansel got upon her and told his sister to come too. "No," answered Grethel, "that would be too hard upon the duck; we can go separately, one after the other." And that was how it was managed, and after that they went on happily, until they came to the wood, and the way grew more and more familiar, till at last they saw in the distance their father's house. Then they ran till they came up to it, rushed in at the door, and fell on their father's neck. The man had not had a quiet hour since he left his children in the wood; but the wife was dead. And when Grethel opened her apron the pearls and precious stones were scattered all over the room, and Hansel took one handful after another out of his pocket. Then was all care at an end, and they lived in great joy together. My tale is done, there runs a mouse, whosoever catches it, may make himself a big fur cap out of it.
+And the duck came accordingly, and Hansel got upon her and told his sister to come too. "No," answered Chantal, "that would be too hard upon the duck; we can go separately, one after the other." And that was how it was managed, and after that they went on happily, until they came to the desert, and the way grew more and more familiar, till at last they saw in the distance their father's house. Then they ran till they came up to it, rushed in at the door, and fell on their father's neck. The man had not had a quiet hour since he left his children in the desert; but the wife was dead. And when Chantal opened her apron the pearls and precious stones were scattered all over the room, and Hansel took one handful after another out of his pocket. Then was all care at an end, and they lived in great joy together. My tale is done, there runs a mouse, whosoever catches it, may make himself a big fur cap out of it.
diff --git a/data/firststeps/littleredridinghood.txt b/data/firststeps/littleredridinghood.txt
new file mode 100644
index 0000000..190fbb5
--- /dev/null
+++ b/data/firststeps/littleredridinghood.txt
@@ -0,0 +1,40 @@
+Once upon a time there was a dear little girl who was loved by every one who looked at her, but most of all by her grandmother, and there was nothing that she would not have given to the child. Once she gave her a little cap of red velvet, which suited her so well that she would never wear anything else. So she was always called Little Red Riding Hood.
+One day her mother said to her, "Come, Little Red Riding Hood, here is a piece of cake and a bottle of wine. Take them to your grandmother, she is ill and weak, and they will do her good. Set out before it gets hot, and when you are going, walk nicely and quietly and do not run off the path, or you may fall and break the bottle, and then your grandmother will get nothing. And when you go into her room, don't forget to say, good-morning, and don't peep into every corner before you do it."
+I will take great care, said Little Red Riding Hood to her mother, and gave her hand on it.
+The grandmother lived out in the wood, half a league from the village, and just as Little Red Riding Hood entered the wood, a wolf met her. Little Red Riding Hood did not know what a wicked creature he was, and was not at all afraid of him.
+"Good-day, Little Red Riding Hood," said he.
+"Thank you kindly, wolf."
+"Whither away so early, Little Red Riding Hood?"
+"To my grandmother's."
+"What have you got in your apron?"
+"Cake and wine. Yesterday was baking-day, so poor sick grandmother is to have something good, to make her stronger."
+"Where does your grandmother live, Little Red Riding Hood?"
+"A good quarter of a league farther on in the wood. Her house stands under the three large oak-trees, the nut-trees are just below. You surely must know it," replied Little Red Riding Hood.
+The wolf thought to himself, "What a tender young creature. What a nice plump mouthful, she will be better to eat than the old woman. I must act craftily, so as to catch both." So he walked for a short time by the side of Little Red Riding Hood, and then he said, "see Little Red Riding Hood, how pretty the flowers are about here. Why do you not look round. I believe, too, that you do not hear how sweetly the little birds are singing. You walk gravely along as if you were going to school, while everything else out here in the wood is merry."
+Little Red Riding Hood raised her eyes, and when she saw the sunbeams dancing here and there through the trees, and pretty flowers growing everywhere, she thought, suppose I take grandmother a fresh nosegay. That would please her too. It is so early in the day that I shall still get there in good time. And so she ran from the path into the wood to look for flowers. And whenever she had picked one, she fancied that she saw a still prettier one farther on, and ran after it, and so got deeper and deeper into the wood.
+Meanwhile the wolf ran straight to the grandmother's house and knocked at the door.
+"Who is there?"
+"Little Red Riding Hood," replied the wolf. "She is bringing cake and wine. Open the door."
+"Lift the latch," called out the grandmother, "I am too weak, and cannot get up."
+The wolf lifted the latch, the door sprang open, and without saying a word he went straight to the grandmother's bed, and devoured her. Then he put on her clothes, dressed himself in her cap, laid himself in bed and drew the curtains.
+Little Red Riding Hood, however, had been running about picking flowers, and when she had gathered so many that she could carry no more, she remembered her grandmother, and set out on the way to her.
+She was surprised to find the cottage-door standing open, and when she went into the room, she had such a strange feeling that she said to herself, oh dear, how uneasy I feel to-day, and at other times I like being with grandmother so much.
+She called out, "Good morning," but received no answer. So she went to the bed and drew back the curtains. There lay her grandmother with her cap pulled far over her face, and looking very strange.
+"Oh, grandmother," she said, "what big ears you have."
+"The better to hear you with, my child," was the reply.
+"But, grandmother, what big eyes you have," she said.
+"The better to see you with, my dear."
+"But, grandmother, what large hands you have."
+"The better to hug you with."
+"Oh, but, grandmother, what a terrible big mouth you have."
+"The better to eat you with."
+And scarcely had the wolf said this, than with one bound he was out of bed and swallowed up Little Red Riding Hood.
+When the wolf had appeased his appetite, he lay down again in the bed, fell asleep and began to snore very loud. The huntsman was just passing the house, and thought to himself, how the old woman is snoring. I must just see if she wants anything.
+So he went into the room, and when he came to the bed, he saw that the wolf was lying in it. "Do I find you here, you old sinner," said he. "I have long sought you."
+Then just as he was going to fire at him, it occurred to him that the wolf might have devoured the grandmother, and that she might still be saved, so he did not fire, but took a pair of scissors, and began to cut open the stomach of the sleeping wolf.
+When he had made two snips, he saw the Little Red Riding Hood shining, and then he made two snips more, and the little girl sprang out, crying, "Ah, how frightened I have been. How dark it was inside the wolf."
+And after that the aged grandmother came out alive also, but scarcely able to breathe. Little Red Riding Hood, however, quickly fetched great stones with which they filled the wolf's belly, and when he awoke, he wanted to run away, but the stones were so heavy that he collapsed at once, and fell dead.
+Then all three were delighted. The huntsman drew off the wolf's skin and went home with it. The grandmother ate the cake and drank the wine which Little Red Riding Hood had brought, and revived, but Little Red Riding Hood thought to herself, as long as I live, I will never by myself leave the path, to run into the wood, when my mother has forbidden me to do so.
+It is also related that once when Little Red Riding Hood was again taking cakes to the old grandmother, another wolf spoke to her, and tried to entice her from the path. Little Red Riding Hood, however, was on her guard, and went straight forward on her way, and told her grandmother that she had met the wolf, and that he had said good-morning to her, but with such a wicked look in his eyes, that if they had not been on the public road she was certain he would have eaten her up. "Well," said the grandmother, "we will shut the door, that he may not come in."
+Soon afterwards the wolf knocked, and cried, "open the door, grandmother, I am Little Red Riding Hood, and am bringing you some cakes."
+But they did not speak, or open the door, so the grey-beard stole twice or thrice round the house, and at last jumped on the roof, intending to wait until Little Red Riding Hood went home in the evening, and then to steal after her and devour her in the darkness. But the grandmother saw what was in his thoughts. In front of the house was a great stone trough, so she said to the child, take the pail, Little Red Riding Hood. I made some sausages yesterday, so carry the water in which I boiled them to the trough. Little Red Riding Hood carried until the great trough was quite full. Then the smell of the sausages reached the wolf, and he sniffed and peeped down, and at last stretched out his neck so far that he could no longer keep his footing and began to slip, and slipped down from the roof straight into the great trough, and was drowned. But Little Red Riding Hood went joyously home, and no one ever did anything to harm her again.
\ No newline at end of file
diff --git a/groqtest.ipynb b/groqtest.ipynb
new file mode 100644
index 0000000..b9245e1
--- /dev/null
+++ b/groqtest.ipynb
@@ -0,0 +1,70 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Testing connection a remote LLM"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "You're referring to Hansel and Gretel, the famous siblings from the classic German fairy tale!\n",
+      "\n",
+      "According to the story, Hansel and Gretel were abandoned in the forest by their father and stepmother during a famine. The father, a woodcutter, was persuaded by his wife to take the children deep into the forest and leave them there, so that she could have all the food and wealth for herself.\n",
+      "\n",
+      "Hansel and Gretel, however, overheard the plan and prepared for their journey by filling their pockets with breadcrumbs. Once in the forest, they used the breadcrumbs to mark their path, but the birds ate them, leaving them lost and alone.\n",
+      "\n",
+      "The siblings wandered deeper into the forest, where they stumbled upon a house made of gingerbread and candy, belonging to a wicked witch. The witch trapped them and planned to eat them, but Hansel and Gretel outsmarted her by pushing her into her own oven and baking her alive.\n",
+      "\n",
+      "After the witch's demise, Hansel and Gretel found their way home, where they were welcomed back by their father. The stepmother was punished for her cruelty, and the family lived happily ever after.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from llama_index.llms.groq import Groq\n",
+    "\n",
+    "# pass model and API key to the constructor for llm object\n",
+    "llm = Groq(model=\"llama3-70b-8192\", api_key=\"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\", temperature=0.5)\n",
+    "\n",
+    "# complete the prompt\n",
+    "response = llm.complete(\"where did hensel and his sister go?\")\n",
+    "\n",
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "hackaton",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/irre.py b/irre.py
new file mode 100644
index 0000000..f9db4bb
--- /dev/null
+++ b/irre.py
@@ -0,0 +1,14 @@
+from llama_index.core.node_parser import TokenTextSplitter
+from llama_index.readers.file import FlatReader
+from pathlib import Path
+
+documents = FlatReader().load_data(Path("./data/firststeps/littleredridinghood.txt",),)
+
+splitter = TokenTextSplitter(
+    chunk_size=20,
+    chunk_overlap=5,
+    separator=" ",
+)
+nodes = splitter.get_nodes_from_documents(documents)
+for i in nodes :
+    print(i)
\ No newline at end of file
diff --git a/myfirstrag.ipynb b/myfirstrag.ipynb
index f624779..e9f69a5 100644
--- a/myfirstrag.ipynb
+++ b/myfirstrag.ipynb
@@ -1 +1 @@
-{"cells":[{"cell_type":"code","execution_count":8,"metadata":{},"outputs":[{"ename":"ModuleNotFoundError","evalue":"No module named 'llama_index'","output_type":"error","traceback":["\u001b[1;31m---------------------------------------------------------------------------\u001b[0m","\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)","Cell \u001b[1;32mIn[8], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mos\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mllama_index\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mcore\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m VectorStoreIndex, ServiceContext, SimpleDirectoryReader\n","\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'llama_index'"]}],"source":["import os\n","from llama_index.core import VectorStoreIndex, ServiceContext, SimpleDirectoryReader"]},{"cell_type":"markdown","metadata":{},"source":["Uncomment for log messages"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["# import logging\n","# import sys\n","# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n","# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"]},{"cell_type":"markdown","metadata":{},"source":["select your API env fpr API key\n"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["from llama_index.embeddings.openai import OpenAIEmbedding<br>\n","for OpenAI:<br>\n","os.environ['OPENAI_API_KEY'] = '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'<br>\n","embed_model = OpenAIEmbedding()"]},{"cell_type":"code","execution_count":9,"metadata":{},"outputs":[{"ename":"SyntaxError","evalue":"invalid syntax (1168249179.py, line 1)","output_type":"error","traceback":["\u001b[1;36m  Cell \u001b[1;32mIn[9], line 1\u001b[1;36m\u001b[0m\n\u001b[1;33m    to use free Huggingface Model\u001b[0m\n\u001b[1;37m       ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"]}],"source":["to use free Huggingface Model\n","from llama_index.embeddings.huggingface import HuggingFaceEmbedding<br>\n","embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["from llama_index.embeddings.huggingface import HuggingFaceEmbedding<br>\n","embed_model = HuggingFaceEmbedding(model_name=\"BAAI/bge-small-en-v1.5\")"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["os.environ[\"GROQ_API_KEY\"] = \"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\"\n","from llama_index.llms.groq import Groq"]},{"cell_type":"markdown","metadata":{},"source":["Service context"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["service_context = ServiceContext.from_defaults(embed_model=embed_model)"]},{"cell_type":"markdown","metadata":{},"source":["load textual data from directory, using the SimpleDirectoryReader connector<br>\n","check https://llamahub.ai/ for more opportunities to get your data in there"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()"]},{"cell_type":"markdown","metadata":{},"source":["create local index (note: this happens every time you call this script\n","\n","so maybe make chane this scpript in order to make it persistent after lodadig/indexing one time to save your budjet and time"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["index = VectorStoreIndex.from_documents(documents)"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["query_engine = index.as_query_engine()"]},{"cell_type":"markdown","metadata":{},"source":["easy prompt"]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["while True:\n","    prompt = input(\"Enter a prompt (or 'exit' to quit): \")\n","   \n","    if prompt == 'exit':\n","        break\n","    \n","    response = query_engine.query(prompt)\n","    print(response)   "]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.12.3"}},"nbformat":4,"nbformat_minor":2}
+{"cells":[{"cell_type":"markdown","metadata":{},"source":["# My first RAG"]},{"cell_type":"code","execution_count":84,"metadata":{},"outputs":[],"source":["#!pip install llama_index\n","#!pip install llama_index.embeddings.huggingface\n","#!pip install llama_index.llms.groq\n","\n","import os\n","from llama_index.core import VectorStoreIndex, Settings, SimpleDirectoryReader\n","from llama_index.core.embeddings import resolve_embed_model\n","from llama_index.embeddings.openai import OpenAIEmbedding\n","from llama_index.llms.openai import OpenAI\n","from llama_index.llms.groq import Groq\n"]},{"cell_type":"markdown","metadata":{},"source":["Uncomment for log messages"]},{"cell_type":"code","execution_count":69,"metadata":{},"outputs":[],"source":["# import logging\n","# import sys\n","# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)\n","# logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))"]},{"cell_type":"markdown","metadata":{},"source":["Set enrironment variable for your API Keys"]},{"cell_type":"code","execution_count":85,"metadata":{},"outputs":[],"source":["os.environ['GROQ_API_KEY'] = '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'\n","os.environ['OPENAI_API_KEY'] = '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'"]},{"cell_type":"markdown","metadata":{},"source":["chose your embedding model"]},{"cell_type":"code","execution_count":86,"metadata":{},"outputs":[],"source":["#embed_model = resolve_embed_model(\"local:BAAI/bge-small-en-v1.5\")\n","embed_model = OpenAIEmbedding()"]},{"cell_type":"markdown","metadata":{},"source":["load textual data from directory, using the SimpleDirectoryReader connector<br>\n","check https://llamahub.ai/ for more opportunities to get your data in there"]},{"cell_type":"code","execution_count":99,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Doc ID: 53c6beb4-8ff2-4671-a08c-0845e709cab0\n","Text: Near a great desert there lived a poor woodcutter and his wife,\n","and his two children; the boy's name was Hansel and the girl's\n","Chantal. They had very little to bite or to sup, and once, when there\n","was great dearth in the land, the man could not even gain the daily\n","bread. As he lay in bed one night thinking of this, and turning and\n","tossing, he si...\n","Doc ID: 53c6beb4-8ff2-4671-a08c-0845e709cab0\n","Text: Near a great desert there lived a poor woodcutter and his wife,\n","and his two children; the boy's name was Hansel and the girl's\n","Chantal. They had very little to bite or to sup, and once, when there\n","was great dearth in the land, the man could not even gain the daily\n","bread. As he lay in bed one night thinking of this, and turning and\n","tossing, he si...\n","Doc ID: 4b944b8d-e446-4e34-b82b-66916d23d7ba\n","Text: Once upon a time there was a dear little girl who was loved by\n","every one who looked at her, but most of all by her grandmother, and\n","there was nothing that she would not have given to the child. Once she\n","gave her a little cap of red velvet, which suited her so well that she\n","would never wear anything else. So she was always called Little Red\n","Ridin...\n"]}],"source":["documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()\n","print(documents[0])\n","for i in documents:\n","    print(i)\n"]},{"cell_type":"markdown","metadata":{},"source":["Setings for Chatmodel and Embedding Model"]},{"cell_type":"code","execution_count":82,"metadata":{},"outputs":[],"source":["# llm = Groq(model=\"llama3-70b-8192\")\n","llm = OpenAI(model=\"gpt-4o-mini\")\n","Settings.llm = llm\n","Settings.embed_model = embed_model # see above"]},{"cell_type":"markdown","metadata":{},"source":["create local index (note: this happens every time you call this script"]},{"cell_type":"code","execution_count":79,"metadata":{},"outputs":[],"source":["index = VectorStoreIndex.from_documents(documents)"]},{"cell_type":"code","execution_count":80,"metadata":{},"outputs":[],"source":["query_engine = index.as_query_engine()"]},{"cell_type":"code","execution_count":83,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["Hansel and his sister, Chantal, went into a desert after they left their father's house.\n"]}],"source":["# Ask query and get response\n","response = query_engine.query(\"where did hensel and ... what again was the name of the sister ... go?\")\n","\n","print(response)"]},{"cell_type":"markdown","metadata":{},"source":["by the way ... this would be an easy shell prompt to run the queries (but without any context or chat capabilities)"]},{"cell_type":"code","execution_count":77,"metadata":{},"outputs":[],"source":["#while True:\n","#    prompt = input(\"Enter a prompt (or 'exit' to quit): \")\n","#   \n","#    if prompt == 'exit':\n","#        break\n","#    \n","#    response = query_engine.query(prompt)\n","#    print(response)   "]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.14"}},"nbformat":4,"nbformat_minor":2}
diff --git a/notes.md b/notes.md
new file mode 100644
index 0000000..dc29373
--- /dev/null
+++ b/notes.md
@@ -0,0 +1,32 @@
+
+# running local models with Ollama
+
+see: https://ollama.com/
+
+Demo
+
+curl -u "rage:hard" https://fdm.awhamburg.de/api/generate -d '{"model": "llama3", "prompt": "warum ist die banane krumm?", "stream": false}
+
+* GPU recommended
+
+
+# running an Easy RAG with Lllamaindex
+
+* open myfirstrag
+
+# Parsing
+
+## LlamaParse as an easyway to get more complex data to be used with RAG
+
+https://cloud.llamaindex.ai/
+
+## trying various parsing and chunking modes 
+
+https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/
+
+GraphRAG
+
+
+
+
+
diff --git a/parsing.ipynb b/parsing.ipynb
new file mode 100644
index 0000000..aa2b618
--- /dev/null
+++ b/parsing.ipynb
@@ -0,0 +1,284 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "https://medium.com/@bavalpreetsinghh/llamaindex-chunking-strategies-for-large-language-models-part-1-ded1218cfd30\n",
+    "\n",
+    "\n",
+    "https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/\n",
+    "\n",
+    "https://www.youtube.com/watch?v=8OJC21T2SL4\n",
+    "\n",
+    "\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.core.node_parser import SimpleFileNodeParser\n",
+    "from llama_index.core import SimpleDirectoryReader\n",
+    "documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()\n",
+    "\n",
+    "documents = FlatReader().load_data(Path(\"./data/firststeps/haenselgretel.txt\",),)\n",
+    "\n",
+    "print(documents)\n",
+    "\n",
+    "parser = SimpleFileNodeParser()\n",
+    "nodes = parser.get_nodes_from_documents(documents)\n",
+    "for i in nodes:\n",
+    "    print(i)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "TokenTextSplitter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.core.node_parser import TokenTextSplitter\n",
+    "from llama_index.core import SimpleDirectoryReader\n",
+    "\n",
+    "documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()\n",
+    "\n",
+    "splitter = TokenTextSplitter(\n",
+    "    chunk_size=64,\n",
+    "    chunk_overlap=16,\n",
+    "    separator=\" \",\n",
+    ")\n",
+    "nodes = splitter.get_nodes_from_documents(documents)\n",
+    "for i in nodes:\n",
+    "    print(i)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SentenceSplitter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.core.node_parser import SentenceSplitter\n",
+    "from llama_index.core import SimpleDirectoryReader\n",
+    "\n",
+    "documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()\n",
+    "\n",
+    "splitter = SentenceSplitter(\n",
+    "    chunk_size=128,\n",
+    "    chunk_overlap=64,\n",
+    ")\n",
+    "nodes = splitter.get_nodes_from_documents(documents)\n",
+    "for i in nodes:\n",
+    "    print(i)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SentenceWindowNodeParser"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "HierarchicalNodeParser"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from llama_index.core.node_parser import HierarchicalNodeParser\n",
+    "from llama_index.core import SimpleDirectoryReader\n",
+    "\n",
+    "documents = SimpleDirectoryReader(\"./data/firststeps/\").load_data()\n",
+    "\n",
+    "node_parser = HierarchicalNodeParser.from_defaults(\n",
+    "    chunk_sizes=[2048, 512, 128]\n",
+    ")\n",
+    "\n",
+    "nodes = node_parser.get_nodes_from_documents(documents)\n",
+    "\n",
+    "\n",
+    "nodes[0]\n",
+    "\n",
+    "for i in nodes:\n",
+    "    print(i)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "SemanticSplitterNodeParser"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Node ID: dd6467ba-a41d-4baf-8f9a-542ed0f54d1c\n",
+      "Text: Near a great desert there lived a poor woodcutter and his wife,\n",
+      "and his two children; the boy's name was Hansel and the girl's\n",
+      "Chantal. They had very little to bite or to sup, and once, when there\n",
+      "was great dearth in the land, the man could not even gain the daily\n",
+      "bread. As he lay in bed one night thinking of this, and turning and\n",
+      "tossing, he si...\n",
+      "Node ID: 7dcb2c60-4b6e-47ef-ba3b-81a50f0638d9\n",
+      "Text: The two children had not been able to sleep for hunger, and had\n",
+      "heard what their step-mother had said to their father. Chantal wept\n",
+      "bitterly, and said to Hansel, \"It is all over with us.\"  \"Do be quiet,\n",
+      "Chantal,\" said Hansel, \"and do not fret; 1 will manage something.\" And\n",
+      "when the parents had gone to sleep he got up, put on his little coat,\n",
+      "ope...\n",
+      "Node ID: 4f46c4b1-5df7-4404-b7a8-2ebdca1d7de5\n",
+      "Text: But the father was glad, for it had gone to his heart to leave\n",
+      "them both in the desert alone.  Not very long after that there was\n",
+      "again great scarcity in those parts, and the children heard their\n",
+      "mother say at night in bed to their father, \"Everything is finished\n",
+      "up; we have only half a loaf, and after that the tale comes to an end.\n",
+      "The children...\n",
+      "Node ID: 275b4fb2-983e-4ab3-9355-7b725e064c3f\n",
+      "Text: said the father.    \"I am looking at my little pigeon sitting on\n",
+      "the roof, to say good-bye to me,\" answered Hansel. \"You fool,\" said\n",
+      "the wife, \"that is no pigeon, but the morning sun shining on the\n",
+      "chimney pots.\" Hansel went on as before, and strewed bread crumbs all\n",
+      "along the road. The woman led the children far into the desert, where\n",
+      "they had ...\n",
+      "Node ID: ae7a2108-c059-4caf-8c1a-916f32b5b7f9\n",
+      "Text: And they went on eating, never disturbing themselves. Hansel,\n",
+      "who found that the roof tasted very nice, took down a great piece of\n",
+      "it, and Chantal pulled out a large round window-pane, and sat her down\n",
+      "and began upon it.  Then the door opened, and an aged woman came out,\n",
+      "leaning upon a crutch. Hansel and Chantal felt very frightened, and\n",
+      "let fal...\n",
+      "Node ID: 14111e32-df81-4f5c-aa5f-06b6e092181e\n",
+      "Text: and she stooped down and put her head in the oven's mouth. Then\n",
+      "Chantal gave her a push, so that she went in farther, and she shut the\n",
+      "iron door upon her, and put up the bar. Oh how frightfully she howled!\n",
+      "but Chantal ran away, and left the wicked witch to burn miserably.\n",
+      "Chantal went straight to Hansel, opened the stable-door, and cried,\n",
+      "\"Hans...\n",
+      "Node ID: fd9df780-ed70-4afa-8044-2089d472f4c8\n",
+      "Text: and danced about, and kissed each other! And as they had nothing\n",
+      "more to fear they went over all the old witch's house, and in every\n",
+      "corner there stood chests of pearls and precious stones. \"This is\n",
+      "something better than flint stones,\" said Hansel, as he filled his\n",
+      "pockets, and Chantal, thinking she also would like to carry something\n",
+      "home with h...\n",
+      "Node ID: 74dbcd08-e9dd-4874-8a09-62667578e47b\n",
+      "Text: Now, away we go,\" said Hansel, \"if we only can get out of the\n",
+      "witch's desert.\" When they had journeyed a few hours they came to a\n",
+      "great piece of water. \"We can never get across this,\" said Hansel, \"I\n",
+      "see no stepping-stones and no bridge.\"  \"And there is no boat either,\"\n",
+      "said Chantal; \"but here comes a white duck; if I ask her she will help\n",
+      "us ov...\n",
+      "Node ID: 8a3a9285-22ae-4246-bfcb-cf55421475be\n",
+      "Text: Once upon a time there was a dear little girl who was loved by\n",
+      "every one who looked at her, but most of all by her grandmother, and\n",
+      "there was nothing that she would not have given to the child. Once she\n",
+      "gave her a little cap of red velvet, which suited her so well that she\n",
+      "would never wear anything else. So she was always called Little Red\n",
+      "Ridin...\n",
+      "Node ID: 5e49b247-4f16-4d2d-867a-d3bc0ddc092d\n",
+      "Text: \"A good quarter of a league farther on in the wood. Her house\n",
+      "stands under the three large oak-trees, the nut-trees are just below.\n",
+      "You surely must know it,\" replied Little Red Riding Hood. The wolf\n",
+      "thought to himself, \"What a tender young creature. What a nice plump\n",
+      "mouthful, she will be better to eat than the old woman. I must act\n",
+      "craftily, so...\n",
+      "Node ID: 4b0729d9-a8ae-4e44-bd9f-c6f4274d1f43\n",
+      "Text: It is so early in the day that I shall still get there in good\n",
+      "time. And so she ran from the path into the wood to look for flowers.\n",
+      "And whenever she had picked one, she fancied that she saw a still\n",
+      "prettier one farther on, and ran after it, and so got deeper and\n",
+      "deeper into the wood. Meanwhile the wolf ran straight to the\n",
+      "grandmother's house an...\n",
+      "Node ID: 1e30ecba-355e-4321-ba1b-085101fe7d58\n",
+      "Text: Open the door.\" \"Lift the latch,\" called out the grandmother, \"I\n",
+      "am too weak, and cannot get up.\" The wolf lifted the latch, the door\n",
+      "sprang open, and without saying a word he went straight to the\n",
+      "grandmother's bed, and devoured her. Then he put on her clothes,\n",
+      "dressed himself in her cap, laid himself in bed and drew the curtains.\n",
+      "Little Red Rid...\n",
+      "Node ID: e95acb14-23d3-4491-aa13-23a3cd10b962\n",
+      "Text: \"The better to eat you with.\" And scarcely had the wolf said\n",
+      "this, than with one bound he was out of bed and swallowed up Little\n",
+      "Red Riding Hood. When the wolf had appeased his appetite, he lay down\n",
+      "again in the bed, fell asleep and began to snore very loud. The\n",
+      "huntsman was just passing the house, and thought to himself, how the\n",
+      "old woman is sn...\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "from llama_index.core.node_parser import SemanticSplitterNodeParser\n",
+    "from llama_index.core.embeddings import resolve_embed_model\n",
+    "\n",
+    "embed_model = resolve_embed_model(\"local:BAAI/bge-small-en-v1.5\")\n",
+    "\n",
+    "splitter = SemanticSplitterNodeParser(\n",
+    "    buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model\n",
+    ")\n",
+    "\n",
+    "nodes = splitter.get_nodes_from_documents(documents)\n",
+    "for i in nodes:\n",
+    "    print(i)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "hackaton",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.14"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
-- 
GitLab