Difference between an OM and a Pet Mod
An OM makes broad changes off a consistent rule (Or small set of rules) with the accommodation for oddities generally being limited to banning things. eg Slaking and Regigigas are extremely powerful if allowed to have non-negative Abilities, and are thus usually banned in metas which would, by their own rules, provide them non-negative Abilities. They are not granted special accommodation to remain legal by virtue of being exempted from the OM's rules. (ie AAA doesn't keep them legal while denying them the core gimmick of the meta -it just bans them)
The short version is an OM can usually be summarized accurately and completely in a single sentence, or at least a single paragraph.
A Pet Mod will usually have a guiding principle ("Make everything OU-viable"), but it will still consist of a long series of specific changes to achieve the vision of the creator or creators. ("Creators" may mean the community, collectively) AbNormal is a good example of a Pet Mod because while the goal of the project is simple -remove the Normal type from the game- there is no simple rule to this. AbNormal would be an OM if it simply said "All instances of Normal typing are replaced by the Fighting type", but since individual Pokemon and moves are converted to a wide variety of types by no clear underlying rule (Beyond "Not allowed to stay Normal"), such that you must read the entire list of changes to have a clear idea of the meta, AbNormal is a Pet Mod.
In contrast with an OM, a Pet Mod cannot be reduced down to a rule or small set of rules to achieve perfect clarity -saying "Everything Normal typed becomes other types" is a clear, consistent rule, but it wouldn't tell you what to expect Boomburst's typing to be, nor Return's, nor Frustration's, all of which are, in fact, different types in AbNormal, never mind that Return and Frustration are virtually the same move in standard play! The rule does not outline the result, in short, it only provides a premise for the potential types of changes the Pet Mod makes.
I'm not staff and do not speak for staff but in my experience this seems to be a reasonably accurate guideline to what constitutes an OM vs what constitutes a Pet Mod.