A comment on my post from Diogenes Laertius, listing the critical marks in use in the 3rd century, drew my attention to the work of the Augustan grammarian Aristonicus of Alexandria. Apparently Friedlander in 1853 published the remains of his work on the critical signs used for the Iliad and Odyssey (Peri Semeion).
Friedlander is online, and the corresponding work on the Odyssey also here. But I couldn’t make head or tail of his edition! Some things are beyond me, and evidently this is one.
This is a pity — a statement of what the critical marks were and how they were used would be nice to have in English.