Image to sound conversion, almost there...?
Posted: Sat Oct 19, 2013 2:48 pm
First check out this program, they have a good example of this:
A nice video showing the effect: http://www.youtube.com/watch?v=W8MCAXhEsy4
Their website: http://photosounder.com/
I was like "hey, this can't be so hard..." and I've gotten some progress but I thought... why not just share with you guys to see if you have any good ideas(code)? Because I'm a bit stuck at the moment, I've mostly been testing all kinds of different stuff (hence the messy code) and now I'm open for ideas
This is the picture you will need to use with my test code (save it as test.png):
http://i.imgur.com/NOHRuGb.png
When the code is executed it plays a PCM wave file it created and it also stores this as "test.wav" in the same directory. This is so it can easily be analyzed with Audacity, it's the perfect tool for debugging the output! It can also refresh it's data automatically when the new file is generated!
My code is not cleaned up, there is a lot of variables and stuff that could be removed, but which for now is kept for easy testing. Feel free to try it out and post improvements! But turn the volume down before you run it!
Explanation of code:
The X of the image equals time while Y equals the frequency.
In my code the lower frequencies are near the top and the highest are near the bottom, the frequency range can be modified with these variables "frequencyFrom" and "frequencyTo".
You can also modify how many milliseconds the image is supposed to play with "wantedWidthInMilliseconds".
Since there are 44100 samples per second it is obvious that each pixel has to consist of several samples, that is why "For i=0 To samplesPerPixel-1" is done for every pixel. And the result of all the samples per X unit are averaged to produce the final "wave position" used to create the wave mixed of the different frequencies.
My first idea was kinda that I had a wave for each Y pixel oscillating at its own frequency and then just add all the used waves together to produce the final output wave. But this doesn't really work correctly, when mixing them their different "sine offset" will easily fuck things up, a more intelligent processing has to be done. That's when I eventually tried to calculate the latest "sine offset" using this weird procedure "getWaveAngle(pSample,sample,amplitude.f)" which by analyzing the two latest samples and the average amplitude of all the frequencies used to make those it tries to determine the offset in "angle". This was tested in the "drawWave()" procedure and seemed to work. But it still produces some noise in my example and my code doesn't seem to handle more advanced images correctly at all.
A nice video showing the effect: http://www.youtube.com/watch?v=W8MCAXhEsy4
Their website: http://photosounder.com/
I was like "hey, this can't be so hard..." and I've gotten some progress but I thought... why not just share with you guys to see if you have any good ideas(code)? Because I'm a bit stuck at the moment, I've mostly been testing all kinds of different stuff (hence the messy code) and now I'm open for ideas
This is the picture you will need to use with my test code (save it as test.png):
http://i.imgur.com/NOHRuGb.png
When the code is executed it plays a PCM wave file it created and it also stores this as "test.wav" in the same directory. This is so it can easily be analyzed with Audacity, it's the perfect tool for debugging the output! It can also refresh it's data automatically when the new file is generated!
My code is not cleaned up, there is a lot of variables and stuff that could be removed, but which for now is kept for easy testing. Feel free to try it out and post improvements! But turn the volume down before you run it!
Explanation of code:
The X of the image equals time while Y equals the frequency.
In my code the lower frequencies are near the top and the highest are near the bottom, the frequency range can be modified with these variables "frequencyFrom" and "frequencyTo".
You can also modify how many milliseconds the image is supposed to play with "wantedWidthInMilliseconds".
Since there are 44100 samples per second it is obvious that each pixel has to consist of several samples, that is why "For i=0 To samplesPerPixel-1" is done for every pixel. And the result of all the samples per X unit are averaged to produce the final "wave position" used to create the wave mixed of the different frequencies.
My first idea was kinda that I had a wave for each Y pixel oscillating at its own frequency and then just add all the used waves together to produce the final output wave. But this doesn't really work correctly, when mixing them their different "sine offset" will easily fuck things up, a more intelligent processing has to be done. That's when I eventually tried to calculate the latest "sine offset" using this weird procedure "getWaveAngle(pSample,sample,amplitude.f)" which by analyzing the two latest samples and the average amplitude of all the frequencies used to make those it tries to determine the offset in "angle". This was tested in the "drawWave()" procedure and seemed to work. But it still produces some noise in my example and my code doesn't seem to handle more advanced images correctly at all.
Code: Select all
EnableExplicit: Define.l
UseJPEGImageDecoder()
UsePNGImageDecoder()
#ToRad = 0.017453
Global format.WAVEFORMATEX
Procedure setWaveFormat(bitsPerSample=16,samplesPerSec=44100)
format\wFormatTag = #WAVE_FORMAT_PCM
format\nChannels = 1 ;mono
format\wBitsPerSample = bitsPerSample ;8/16
format\nSamplesPerSec = samplesPerSec ;8000 Hz, 11025 Hz, 22050 Hz, and 44100 Hz
format\nBlockAlign = (format\nChannels * format\wBitsPerSample) / 8 ;equal to the product of nChannels and wBitsPerSample divided by 8 (bits per byte).
format\nAvgBytesPerSec = format\nSamplesPerSec * format\nBlockAlign ;equal to the product of nSamplesPerSec and nBlockAlign
EndProcedure
Procedure addWaveHeader(*address,dataSize,channels,samplesPerSec,blockAlign,bitsPerSample)
; RIFF Chunk
PokeL(*address+ 0, 'FFIR')
PokeL(*address+ 4, dataSize + 36)
PokeL(*address+ 8, 'EVAW')
; FORMAT Chunk
PokeL(*address+ 12, ' tmf')
PokeL(*address+ 16, $0010)
PokeW(*address+ 20, $01)
PokeW(*address+ 22, channels)
PokeL(*address+ 24, samplesPerSec)
PokeL(*address+ 28, samplesPerSec * channels * (bitsPerSample / 8) )
PokeW(*address+ 32, blockAlign)
PokeW(*address+ 34, bitsPerSample)
; DATA Chunk
PokeL(*address+ 36, 'atad')
PokeL(*address+ 40, dataSize)
EndProcedure
Procedure.l sineWave(position,amplitude,frequency)
Protected result, angle.f
angle = (360 / format\nSamplesPerSec) * position * frequency
result = Sin(Radian(angle))*amplitude
ProcedureReturn result
EndProcedure
Procedure.l sineWave2(NA,amplitude,frequency) ;correct if called format\nSamplesPerSec per second
Protected result, multiplier.f
Static angle.f
multiplier = (360 / format\nSamplesPerSec) * frequency
angle + multiplier
If angle > 360.0
angle - 360.0
EndIf
result = Sin(Radian(angle))*amplitude
ProcedureReturn result
EndProcedure
Procedure.f getWaveAngle(pSample,sample,amplitude.f) ;possible buggy, made in a hurry to try it
Protected waveAngle.f; = 0
If pSample > sample
waveAngle = (-90 * (sample / amplitude)) + 180
Else
waveAngle = ( 90 * (sample / amplitude)) + 360
If waveAngle > 360
waveAngle - 360
EndIf
EndIf
ProcedureReturn waveAngle
EndProcedure
Procedure.l test()
Protected *memory, id, size, duration
duration = 1000
size = Int((format\nSamplesPerSec/1000)*duration) * (format\wBitsPerSample / 8)
*memory = AllocateMemory(44+size)
Protected x,y
For x=0 To format\nSamplesPerSec-1 ;1 second
y = sineWave(x,30000,450)
PokeW(*memory+44+(x*2),y)
Next
addWaveHeader(*memory,size,format\nChannels,format\nSamplesPerSec,format\nBlockAlign,format\wBitsPerSample)
id = CatchSound(#PB_Any,*memory)
FreeMemory(*memory)
ProcedureReturn id
EndProcedure
Procedure.l generateFromImage(image$)
Protected height,width,x,y,color, frequency, fMultiplier.f, amplitude
Protected frequencyFrom,frequencyTo
Protected totalSamples, samplesPerPixel, wantedWidthInMilliseconds = 4000, actualWidthInMilliseconds
Protected image, i, sample, differentFrequencies, mSize, mPos, *memory
Protected result, angleMultiplier.f, waveAngle.f, avgSample, lastWaveAngle.f, avgAmplitude.f, pSampleWritten
frequencyFrom = 60
frequencyTo = 5000 ;17000 ;http://www.youtube.com/watch?v=h5l4Rt4Ol7M
image = LoadImage(#PB_Any,image$)
If image
width = ImageWidth(image)
height = ImageHeight(image)
;Debug height
fMultiplier = (frequencyTo-frequencyFrom) / height
samplesPerPixel = Round((format\nSamplesPerSec*(wantedWidthInMilliseconds*0.001))/width,#PB_Round_Up)
totalSamples = samplesPerPixel * width
actualWidthInMilliseconds = Int((totalSamples / format\nSamplesPerSec) * 1000)
mSize = (totalSamples * (format\wBitsPerSample / 8)) + 44
*memory = AllocateMemory(mSize)
mPos = 44
Dim sampleTotal(samplesPerPixel-1)
Dim angleMultiplier.f(height-1)
Dim waveAngle.f(height-1)
Dim waveAngleTotal.f(samplesPerPixel-1)
Dim amplitudeTotal.f(samplesPerPixel-1)
For y=0 To height-1
frequency = (y * fMultiplier) + frequencyFrom
angleMultiplier(y) = (360 / format\nSamplesPerSec) * frequency
Next
StartDrawing(ImageOutput(image))
For x=0 To width-1
For y=0 To height-1
color = Point(x,y)
If color <> #Black
differentFrequencies + 1
amplitude = Int(Red(color)*(32767/255))
waveAngle(y) = getWaveAngle(pSampleWritten,avgSample,avgAmplitude);lastWaveAngle ;last average one
For i=0 To samplesPerPixel-1
waveAngle(y) + angleMultiplier(y)
If waveAngle(y) > 360.0
waveAngle(y) - 360.0
EndIf
sample = Sin(Radian(waveAngle(y)))*amplitude
sampleTotal(i) + sample ;all y pixels adds their samples so an average can be calculated
waveAngleTotal(i) + waveAngle(y)
amplitudeTotal(i) + amplitude
Next
Else ;still change their wave pos
If 0 ;0 means turned off for now
;waveAngle(y) = lastWaveAngle ;last average one
For i=0 To samplesPerPixel-1
waveAngle(y) + angleMultiplier(y)
If waveAngle(y) > 360.0
waveAngle(y) - 360.0
EndIf
Next
EndIf
EndIf
Next
For i=0 To samplesPerPixel-1
pSampleWritten = avgSample
avgSample = Int(sampleTotal(i)/differentFrequencies)
waveAngle = waveAngleTotal(i)/differentFrequencies
avgAmplitude = amplitudeTotal(i)/differentFrequencies
lastWaveAngle = waveAngle
sampleTotal(i) = 0 ;reset
waveAngleTotal(i) = 0 ;reset
amplitudeTotal(i) = 0
PokeW(*memory+mPos,avgSample)
mPos + 2
Next
differentFrequencies = 0
Next
StopDrawing()
addWaveHeader(*memory,mSize-44,format\nChannels,format\nSamplesPerSec,format\nBlockAlign,format\wBitsPerSample)
If CreateFile(0,"test.wav")
WriteData(0,*memory,mSize)
CloseFile(0)
EndIf
result = CatchSound(#PB_Any,*memory)
FreeMemory(*memory)
Else
Debug "error"
EndIf
ProcedureReturn result
EndProcedure
Procedure drawWave()
Protected window, imageGadet, i, signal, image
window = OpenWindow(#PB_Any,0,0,1200,400,"",#PB_Window_ScreenCentered|
#PB_Window_SystemMenu|
#PB_Window_TitleBar)
imageGadet = ImageGadget(#PB_Any,0,0,1200,400,0)
image = CreateImage(#PB_Any,1200,400,24,#Black)
StartDrawing(ImageOutput(image))
Line(0,100,1200,1,#Gray)
Line(0,200,1200,1,#Gray)
Line(0,300,1200,1,#Gray)
Protected waveAngle.f, angleMultiplier.f, frequency, counter, pSignal
For i=0 To 1200-1
frequency = 100
angleMultiplier = (360 / format\nSamplesPerSec) * frequency
waveAngle + angleMultiplier
If waveAngle > 360.0
waveAngle - 360.0
EndIf
signal = Sin(Radian(waveAngle))*100
counter+1
If counter = 20
counter = 0
DrawText(i,signal+200,Str(getWaveAngle(pSignal,signal,100)),#Green) ;debugging the predicted angle
EndIf
pSignal = signal
;signal = sineWave(i,100,50)
Plot(i,signal+200,#Green)
;signal = sineWave2(i,100,50)
;Plot(i,signal+200,#Red)
Next
StopDrawing()
SetGadgetState(imageGadet,ImageID(image))
Repeat
Until WaitWindowEvent() = #PB_Event_CloseWindow
EndProcedure
InitSound()
setWaveFormat(16,44100)
Define sound
Select 0
Case 0
sound = generateFromImage("test.png")
PlaySound(sound)
Delay(4200)
Case 1
drawWave()
Case 2
sound = test()
PlaySound(sound)
Delay(2000)
EndSelect
