The future of fishing is big data and artificial intelligence

New England’s groundfish season is in full swing, as hundreds of dayboat fishermen from Rhode Island to Maine take to the water in search of the region’s iconic cod and haddock. But this year, several dozen of them are hauling in their catch under the watchful eye of video cameras as part of a new effort to use technology to better sustain the area’s fisheries and the communities that depend on them.

Video observation on fishing boats — electronic monitoring — is picking up steam in the Northeast and nationally as a cost-effective means to ensure that fishing vessels aren’t catching more fish than allowed while informing local fisheries management. While several issues remain to be solved before the technology can be widely deployed — such as the costs of reviewing and storing data — electronic monitoring is beginning to deliver on its potential to lower fishermen’s costs, provide scientists with better data, restore trust where it’s broken and ultimately help consumers gain a greater understanding of where their seafood is coming from.

“Electronic monitoring is a tremendous tool,” says Brett Alger, national electronics technology coordinator for NOAA Fisheries. “It isn’t necessarily for everyone or every fishery,” but “we’re working collaboratively in all of our regions with fishermen on the ground to understand their needs. I expect it to grow.”

The technology is required for highly migratory longline species in the Atlantic (swordfish). It’s thriving in the Pacific coast groundfish industry, and dozens of other fisheries regions have pilot initiatives.

“I was dead set against electronic monitoring when it first started,” says Nick Muto, a veteran commercial fisherman based in Cape Cod. “I wasn’t in favor of having Big Brother over my shoulder all the time. There was a lot of gray area about what we were calling data. There were concerns about confidentiality and who would have access to the data.”

Once the fishery’s stakeholders were able to work through those issues; however, Muto decided to give it a go. Last year he installed cameras on his 45-foot vessel, the Dawn T. He opted to try the technology because he says he “was upset with the broken system of human monitors.”

Human observers are widely used to monitor catch in quota-managed fisheries, and they’re expensive: It costs roughly $700 a day for an observer in New England.

New England uses a “catch shares” system to regulate the groundfish fishery. Fishermen are allotted permits for the total amount of fish they can catch in one year and discards (e.g., juveniles) count toward that quota. Human observers serve as monitors to determine whether fishermen are accurately reporting their discards at sea. Fishermen are required to notify regulators about their trips 48 hours before they leave the dock.

While human observers are required this year on only 15 percent of trips under the Northeast’s At-Sea Monitoring (ASM) Program, the costs add up in an industry with very tight margins. Many fishermen also dislike having another person on board a small vessel.

“It’s an unsafe situation,” says Muto, a first-generation fisherman who got his start as a deckhand, cleaning lobster pots and sorting gear. “Sure, an observer has insurance, but on top of all the other headaches, I now have responsibility for this other person, with all their scales and baskets. It makes a small boat even smaller,” he adds.

Regulators began allowing cameras on New England’s dayboats (smaller fishing vessels that typically spend only a few days out at sea) as a substitute for human observers in 2016. Two other programs are still in the pilot phase in New England, including one for larger groundfish trawlers and one for the herring fishery.

Muto’s vessel was outfitted with cameras, at a cost of about $8,000, through a collaborative venture between NOAA’s regional office and science center, The Nature Conservancy (TNC), the Gulf of Maine Research Institute and the Cape Cod Commercial Fishermen’s Alliance. Camera costs are currently subsidized by NOAA Fisheries and its partners.

The cameras run the entire time Muto and his crew are out on the water. They record how the fishermen handle their discards, the fish they’re not allowed to keep because of size or species type, but that count toward their quotas. The cost is lower than what he’d pay for an in-person monitor.

The biggest cost of electronic monitoring, however, is the labor required to review the video. Someone has to watch all of the footage collected from two to four days out at sea, or even longer for the larger vessels. McGuire said that because the technology is still evolving, costs are higher than they would be after broad adoption; he noted that the cost of video review alone is between $300 and $350 per day.

One way to cut those costs is to spot-check the video footage and compare it to fishermen’s reports, in a “trust but verify” system, says Christopher McGuire, marine program director for TNC in Massachusetts. That approach is working in other fisheries, such as British Columbia’s groundfish fishery, where regulators view about 10 percent of the footage.

Using machine learning and artificial intelligence

Another way to cut costs is to use computers to review the footage. McGuire says there’s been a lot of talk about automating the review, but the common refrain is that it’s still five years off.

To spur faster action, TNC last year spearheaded an online competition, offering a $50,000 prize to computer scientists who could crack the code — that is, teach a computer how to count fish, size them, and identify their species.

“We created an arms race,” says McGuire. “That’s why you do a competition. You’ll never get the top minds to do this because they don’t care about your fish. They all want to work for Google, and one way to get recognized by Google is to win a few of these competitions.”

The contest exceeded McGuire’s expectations. “Winners got close to 100 percent in count and 75 percent accurate on identifying species,” he says. “We proved that automated review is now. Not in five years. And now all of the video-review companies are investing in machine learning.” It’s only a matter of time before a commercial product is available, McGuire believes.

Big Brother or better trust?

With the rise of surveillance in society, from dashboard cameras to Facebook, it’s no surprise that fishermen may be leery about how the video footage that’s collected on their boats will be used.

“The biggest concern is that as soon as you give up your data to a third party, you’re vulnerable immediately. Your data could be used for unintended purposes,” says Jungwoo Ryoo, professor of information sciences and technology at Pennsylvania State University, on the larger trend. But, he says, “blind denial of these useful technologies is not the way to go. Big data helps us do our jobs more efficiently.”

Ryoo cautions fishermen to make sure that privacy and security are in the terms and conditions of electronic monitoring agreements, and to seek protections as a group. Fortunately, fishermen have group power. Electronic monitoring program details are typically worked out by local fishery councils where fishermen have a voice. That’s why New England industry experts think that electronic monitoring could actually help to reverse legendary mistrust between fishermen and scientists.

Fishermen are required to record everything they catch and discard. However, “Scientists don’t typically trust what fishermen write on their trip reports,” says McGuire. “There are a lot of incentives for people to misreport. Even though fisherman are required to fill them out, the management and science largely don’t use them.”

“Electronic monitoring is trying to flip that on its head,” he says. “A fisherman’s catch record would be accepted as an accurate accounting of the trip unless proven otherwise. It’s one of the first ways you start to change the dynamic in the fishery.”

Muto agrees. “We look at the scientists and say, ‘you’re full of shit,’ and they look at us and say the same thing,” he says. But now that he has video footage to point to, he adds, “when I show up to a regulatory meeting or a hearing, I can say, ‘here’s what I’m seeing on the water and here is my evidence to prove it.’”

Electronic monitoring also provides scientists and regulators with a more timely, more accurate assessment of the fishery’s health. And that makes it easier to link catch limits more closely to actual populations and improve the effectiveness of conservation restrictions.

Connecting to consumer traceability

As for consumers, one day, video footage of fishing-vessel activity could be connected to traceability movements that seek to ensure that seafood is ethically and legally harvested.

Ecotrust Canada, the video review partner for New England’s program, is piloting the first such traceability program. ThisFish allows consumers to enter a code online that tells them who caught their fish, how and where, and possibly even watch an online clip. So far it’s focused on Canadian fisheries.

McGuire thinks that the tuna industry is ripe for more of this sort of deck-to-dish traceability because of its high value. But he cautions that it may take a while to catch on. “Electronic monitoring is tough to explain in the three seconds it takes people to walk past the seafood case.”

A future without human observers?

Thus far, Muto says he’s fairly satisfied with how the cameras are working. Recently, he was rewarded when regulators granted greater flexibility to the fishermen using electronic monitoring. Now he can respond to what he sees on the water, and shift from fishing for groundfish to fishing for Atlantic Bluefin tuna without having to first return to port and notify regulators.

Alger says that sort of flexibility appeals to fishermen. “A lot of industry groups are starting to see electronic monitoring as a way to go back to the way it was: ‘I’m just going to go fishing.’ They’re seeing electronic monitoring as a way to be more opportunistic when they go fishing, and to provide more info about what’s occurring in the ocean.”

While today, only a fraction of New England dayboat fishermen are using cameras, McGuire is hopeful that the industry will reach a tipping point where use of the technology becomes the norm. “Fishermen talk a lot,” he says. “We just need one or two in every port. Others will find out.”

A version of this story originally appeared on Civil Eats.