Police in New South Wales pursue more than 80% of Indigenous people found with small amounts of cannabis through the courts while letting others off with warnings, forcing young Aboriginal people into a criminal justice system that legal experts say “they will potentially never get out of”.

The Guardian can reveal that between 2013 and 2017 the police disproportionately used the justice system to prosecute Indigenous people, despite the existence of a specific cautioning scheme introduced to keep minor drug offences out of the courts.

During the five year period, 82.55% of all Indigenous people found with a non-indictable quantity of cannabis were pursued through the courts, compared with only 52.29% for the non-Indigenous population, the data compiled by the NSW Bureau of Crime Statistics and Research shows.

The data shows police were four times more likely to issue cautions to non-Indigenous people. In the five years to 2017, only 11.41% of Indigenous Australians were issued with cannabis cautions compared with 40.03% of the non-Indigenous population.

Karly Warner, the chief executive of the Aboriginal Legal Service, said the data was an illustration of how Indigenous people were treated differently “at every stage” of the justice system. She cited a recent case in which a young Indigenous boy had been caught with a small amount of cannabis and, despite having no prior convictions, had been charged by police.

“As annoyed and as angry as I am about that, I am also not surprised,” she said.

“As this data shows these are not one-offs. This is a complete failure to address systematic racism, perpetrated by individuals, and it is forcing, absolutely forcing, Aboriginal kids, young people and adults into contact with a criminal justice system that they will potentially never get out of again.”

Court data shows that 92.85% of Indigenous Australians taken to court for the cannabis possession charge were either found guilty by a judge or magistrate or pleaded guilty.

While non-Indigenous people had a similar conviction rate, evidence suggests Indigenous people receive harsher sentences. A global report prepared for Penal Reform International used Bocsar data from 2013 to 2017 to find Indigenous women were more likely to receive harsher prison sentences for drug possession offences, “including triple the rate of prison sentences”.

Prof Thalia Anthony, an expert on Indigenous criminalisation at the University of Technology, Sydney, has found similarly disproportionate court action against Indigenous Australians for traffic offences such as driving unlicensed or driving unregistered.

Anthony said the trend revealed more than just a systemic bias against Indigenous Australians. She believes it reflects a conscious belief by police that they need to “add that extra layer of punishment” for Indigenous Australians.

“I absolutely think there is systemic bias, but I think these examples are ones of overt racism,” Anthony told the Guardian.

“When you’ve got the same offence and you’re giving a court notice or even an incarceration penalty to an Indigenous person, that’s just overt racism.”

Anthony said the 437 deaths in custody since the 1991 Royal Commission tracked by Guardian Australia’s Deaths Inside investigation were “the tip of a huge iceberg of disadvantage and disempowerment that criminalisation brings into effect”. 

“For example, we know if a young person is diverted [from the courts], they are much more likely to go on to have more fulsome family lives, they’re more likely to get education and training, they’re more likely to get a job. So all of their indicators are improved if they are diverted,” she said.

The Guardian’s investigation also revealed significant failures in the way Australian law enforcement reports on its policing of Indigenous Australians and other vulnerable groups.

While data on the policing of Indigenous Australians is available in NSW through Bocsar, in other states academics and lawyers say they can see only glimpses of how police interact with minority groups.

In February, Guardian Australia revealed Aboriginal drivers in Western Australia received 3.2 times more fines from being pulled over by police than non-Aboriginal drivers , but slightly fewer penalties from traffic cameras.

The data came from an internal police briefing note which warned the “notable ethnic disparity” was a “clearly disturbing” trend, but was made public only after Guardian Australia obtained it through freedom of information laws.

Similarly disturbing policing trends have been unearthed in other jurisdictions. In 2017, University of Queensland Prof Tamara Dean obtained data from the state’s justice department which showed Indigenous Australians were dramatically over-represented in the enforcement of the state’s controversial public nuisance laws.

Only 3.6% of Queensland’s population is Indigenous, but in the 10 years after the offence was introduced in 2004, about one in four of all adults and more than 40% of children charged with public nuisance offences were Indigenous.

The NSW cannabis cautioning scheme was introduced in 2000 following the state’s landmark drug summit. The NSW Police website says the scheme was introduced in response to the summit’s finding that “arresting people for minor drug offences is not always an effective response”.

Instead of a charge or fine, the caution allows officers to refer people to the Alcohol and Drug Information Service which provides “dedicated, confidential service to a cautioned offender that includes information about treatment, counselling and support services”.

But police still have the discretion to charge people. In a statement the NSW Police pointed to eligibility restrictions around the cannabis cautioning scheme as an explanation for the dramatically different outcomes for Indigenous and non-Indigenous Australians.

For example, officers can only issue two cautions to the same person and the scheme is not available to people who have been previously convicted of a drug offence, sexual offence or any offence involving violence.

“The Cannabis Cautioning Scheme is an important part of the NSW Police Force’s illicit drug strategy,” the statement said.

“Police are encouraged to exercise discretion and issue a caution to eligible offenders. The statistics are a reflection of the ability for an individual to meet the criteria.”