Concise Bible Dictionary:

This refers to the Hebrew letter yod, the smallest letter in the language (Matt. 5:1818For verily I say unto you, Till heaven and earth pass, one jot or one tittle shall in no wise pass from the law, till all be fulfilled. (Matthew 5:18)). The word used is iota, which is the Greek equivalent for the same letter.

Strong’s Dictionary of Greek Words:

of Hebrew origin (the tenth letter of the Hebrew alphabet); "iota", the name of the eighth letter of the Greek alphabet, put (figuratively) for a very small part of anything
KJV Usage:

From Manners and Customs of the Bible:

There may be allusion here to the great care taken by the copyists of the law to secure accuracy even to the smallest letters, or curves or points of letters. ‘lima, “jot,” refers to the yodh,י , the smallest letter in the Hebrew alphabet; κεραία, “tittle,” is an apex or little horn, and refers to the horn-like points which are seen on Hebrew letters, for example, ב, ד, ח, ה, ו, It is worthy of remark that the yodh has one of these points, and the meaning of the text may be, “Not even a yodh, nor the point of a yodh.” The text under consideration is sometimes cited to prove that, in the time of Christ, copies of the law were written in the “square character.”
Sometimes curved extensions resembling horns are attached to the letters by the copyists for ornamentation. Prof. Hackett found in one of the synagogues at Safet a scribe engaged in making a copy of the law. He says: “A more elegant Hebrew manuscript, a more perfect specimen of the calligraphic art, I never saw than that executed by this Jewish amanuensis. No printed page could surpass it in the beauty, symmetry, and distinctness with which the characters were drawn. One peculiarity that struck me at once, as I cast my eye over the parchment, was the horn-like appearance attached to some of the letters. I had seen the same mark before this in Hebrew manuscripts, but never where it was so prominent as here. The sign in question, as connected with the Hebrew letter Lamedh [ל] in particular, had almost the appearance of an intentional imitation of a ram’s head” (Illustrations of Scripture, p. 225).
Dr. Ginsburg, in Kitto’s Cyclopedia, s. v., Jot and Tittle, expresses the opinion that the “tittle” refers to certain small ornaments which the Talmudists were accustomed to place upon the tops of letters. They attached great importance to these ornaments, though they formed no special part of the letters.