Via ferry, the only possible way is to first take a ferry on Tallink-Silja from Riga to Stockholm, and then another ferry from Stockholm to Helsinki. A faster alternative is to take a bus or drive 4-5 hours from Riga to Tallinn, then take one of many ferries to Helsinki (two hours).
Explore Other Ferries